NEPC Resources on Vouchers
Are Private Schools Really Better?
Reviews Worth Sharing: An Impact Analysis of North Carolina's Opportunity Scholarship Program on Student Achievement (June 2018)
Normally, when somebody hears about an evaluation of an education program, they reasonably assume the evaluation will tell them whether the program is working or not. When reading an evaluation report, policymakers, parents, and educators hope the evaluation will tell them if the program is helping the participating students. These seem like obvious, uncontroversial points.
On Monday, June 4, researchers from NC State released a working paper, An Impact Analysis of North Carolina’s Opportunity Scholarship Program on Student Achievement, which purports to be an evaluation of North Carolina’s largest private school voucher program. The authors enthusiastically publicized and distributed the report, making sure to provide advance copies to media organizations and a pro-voucher advocacy group. The report has been highlighted by all of the state’s major media outlets.
But there’s a problem: the report fails to tell us whether the Opportunity Scholarship program is working. The researchers’ efforts tell us nothing about whether accepting an Opportunity Scholarship will help or harm a student’s education.
NEPC Review: Is Public Schooling a Public Good? An Analysis of Schooling Externalities (Cato Institute, May 2018)
A report from the Cato Institute opens with Horace Mann’s well-known conviction that public schools are the bedrock of a democratic society – a public good that should be made available to all. Yet the report, Is Public Schooling a Public Good? An Analysis of Schooling Externalities, improperly conflates the civic and economic definitions of a public good. Although the report begins with Mann’s vision of the role of public schools as building a better society, it then misleadingly shifts the analysis to the economic value of public schools as a market-based “good” like steel or corn. The report relies on a false equivalence of the civic and economic definitions of a public good to advance a proposal for de-funding public schools and introducing a nationwide education savings account (voucher) program. While there is extensive research on the educational purposes of schooling, the Cato report’s limited review of this literature consistently misrepresents the meaning, scope and implications of this literature. The result is a portrayal of public schools as “agents of harm” for what appears to be an ideologically driven thought experiment. Even for those who might be in favor of vouchers, the report’s imbalance, flawed logic and limited research base render the report of no use to policymakers.
NEPC Review: Do Impacts on Test Scores Even Matter? (American Enterprise Institute, March 2018)
A report from the American Enterprise Institute (AEI), Do Impacts on Test Scores Even Matter? Lessons from Long-Run Outcomes in School Choice Research, examines whether student achievement scores on math and English language arts tests align with “long-run” attainment outcomes such as high school graduation rates, college enrollment, and college graduation rates. Drawing on a systematic review of the literature, it concludes that the impacts of school choice programs on test scores are not well connected to such attainment outcomes, which are presented as more positive. This review considers two issues with the report: consistency and evidence. Regarding consistency, the report’s suggestion that achievement scores should play a smaller role in determining the efficacy of school choice models represents a stunning effort to move the goalposts in search of new justifications for supporting their preferred policies. After decades of pro-school-choice research and advocacy promoting test score comparisons with public schools as the primary measurement for evaluating school choice models (e.g., charters and school vouchers), the AEI report now suggests that less attention be given to these learning outcomes. Regarding evidence, the AEI report is riddled with numerous internal inconsistencies in its discussion and treatment of a set of studies that were selected by questionable methods. In view of the 180-degree turn based on questionable evidence, the report — despite the authors’ assertions — is of little use to policymakers.
NEPC Review: Why Indiana Parents Choose: A Cross-Sector Survey of Parents' Views in a Robust School Choice Environment (EdChoice, September 2017)
In this report, a school choice advocacy group presents results from its survey of K-12 parents within and across the public and private sectors. They report that parents are highly satisfied with voucher and tax credit scholarship programs and suggest that the findings support the expansion of school choice programs. However, these and other findings are consistent with research showing general parental satisfaction with their children’s schools. One underemphasized finding is that substantial proportions of public school families—the largest constituency of K-12 parents in Indiana—did not participate in private school choice programs because they are happy with their current schools and want to support public schools. The survey and analysis fall short in four ways. First, three incompatible data collection methods were used to collect small samples of non-representative groups of Indiana parents. Second, the statistical analyses are too weak to draw clear conclusions. Third, while organized like a conventional research study, the report appears to be designed to advance an agenda rather than provide substantive answers to important policy questions. Finally, the report provides little new information about parents’ experiences with their children’s schools. Thus, the report does not add to our knowledge about school choice in Indiana or provide much useful information about public support for school choice programs.
NEPC Review: The Tax-Credit Scholarship Audit: Do Publicly Funded Private School Choice Programs Save Money? (EdChoice, October 2016)
This report asserts that tax credit scholarship programs, that distribute scholarships to students via Scholarship Tuition Organizations (STOs), have saved state treasuries between $1.7 and $3.4 billion dollars since 1998. The report argues that these programs are able to realize fiscal savings as a result of students leaving public schools and entering private schools (defined as “switchers”). The report claims that the percentage of students leaving public schools, coupled with the offset of variable per-student costs that districts no longer need to expend, have resulted in the sizable financial savings for state governments. This review questions the method used to estimate the percentage of switcher students across these various programs, and examines how the report determines variable cost fluctuations for each student that leaves public schooling. Since no STO programs require officials to track data on which students transfer out of public schooling into private, these lax accountability standards have led the report author to estimate fiscal savings using conjecture. Instead of following students, they interpreted broad population changes to STOs. Consequently, the results of this report do not provide an acceptable causal conclusion for policymakers. Suggestions for more extensive accounting procedures along with more nuanced methodologies for calculating true variable student costs are discussed.
NEPC Review: Apples to Apples: The Definitive Look at School Test Scores in Milwaukee and Wisconsin (Wisconsin Institute for Law and Liberty, March 2017)
The report reviewed here compares Wisconsin student test score performance for the 2015-16 school year across public schools, charter schools and private schools participating in one of the state’s voucher programs. Comparing a single year’s test scores across school sectors that serve different student populations is inherently problematic. The report uses linear regression models to attempt to adjust for these differences and make what the authors claim are “apples to apples” comparisons. Based on these comparisons, the report concludes that charter schools and private schools participating in the voucher programs are more effective than traditional public schools. Unfortunately, the limited nature of available data undermines any such causal conclusions. The inadequate and small number of school-level variables included in the regression models are not able to control for important confounding variables, most notably prior student achievement. Further, the use of aggregate percent proficient metrics masks variation in performance across grade levels and makes the results sensitive to the (arbitrary) location of the proficiency cut scores. The report’s description of methods and results also includes some troubling inconsistencies. Thus, while the report does present important descriptive statistics about test score performance in Wisconsin, it cannot provide answers for those interested in determining which schools or school choice policies in Wisconsin are most effective.
NEPC Review: Squeezing the Public School Districts: The Fiscal Effects of Eliminating the Louisiana Scholarship Program (University of Arkansas Department of Education Reform, August 2016)
Two new papers from researchers at the University of Arkansas predict the budgetary consequences of terminating the Louisiana Scholarship Program (LSP), a voucher program that funds over 7,100 Louisiana students to attend private schools. Using an economic model, the papers offer several different scenarios and then conclude that terminating the LSP would increase the costs statewide and do so in almost all districts in the state. The papers’ findings are reasonable but do not make a fully convincing case that the state will incur extra expenditures without the LSP. There may be savings or additional expenditures, depending on several key parameters which have not been precisely estimated. Puzzlingly, the reports’ findings of extra costs run counter to those of the state’s Legislative Fiscal Office, but the reports do not mention this contrary evidence. In any case, the net fiscal effect of terminating the LSP is unlikely to be large; in the context of a $9 billion state expenditure, the change would likely be less than $10 million.
NEPC Review: A Win-Win Solution: The Empirical Evidence on School Choice (Friedman Foundation for Educational Choice, May 2016)
Two reports claim to offer empirical support for the efficacy of voucher programs that allow parents to use taxpayer dollars to send their children to private schools. One report (A Win-Win Solution) is the latest in a series from the Friedman Foundation for Educational Choice. The Friedman report reviews studies purporting to show positive impacts from voucher programs in the US. The other report (The Participant Effects of Private School Vouchers across the Globe) is from the Department of Education Reform at the University of Arkansas. The authors of the Arkansas report conducted a limited meta-analysis of US and international studies of voucher programs. The two reports share a positive view of the impacts of vouchers, and both focus on randomized studies of those effects. Both reports are marred by a number of serious problems and errors, including misrepresentations of the research literature, a failure to acknowledge the limitations of their approaches, not addressing the shortcomings of the theoretical underpinnings of vouchers, and the use of methods that bias the selections of the studies they utilize. The Friedman report is a rudimentary “vote-counting “analysis of an extremely narrow set of 18 studies using a biased counting system. The Arkansas meta-analysis aspires to be “global,” but despite identifying over 9,000 potential studies for the analysis, ultimately uses only 19, almost half of which were conducted by the Arkansas authors or their associates. Moreover, the “global” meta-analysis only encompasses three countries (one of which is consistently misspelled). Together, their manifold serious flaws undercut the trustworthiness and usefulness of these reports.
NEPC Review: The School Choice Voucher: A "Get Out of Jail" Card? (University of Arkansas Department of Education Reform, March 2016)
In a report, University of Arkansas researchers studied the relationship between school vouchers and crime. Using data from the Milwaukee Parental Choice Program (MPCP), they compared crimes processed through the Wisconsin courts for program participants and a matched sample of public school students. Based on a series of comparisons, it finds that some groups of MPCP students are less likely to commit crimes as adults. This result is plausible: education and crime are often found to be negatively correlated and the MPCP has generated some, albeit modest and mixed, benefits. However, the study’s title should not imply that voucher programs are a "get-out-of-jail" card, and the evidence in the study is not causal. One concern is that the paper employs a matching method that omits some important factors that explain school choice and crime. Also, the results are highly variable, with most of the association between MPCP participation and measures of adult crime showing statistically insignificant results. Indeed, a valid interpretation of the paper is that vouchers and crime are not correlated. Conversely, for subgroups and estimation approaches that do yield statistically significant associations, the MPCP effects appear to be too extreme. Even assuming that vouchers do reduce adult crime, it remains unclear by what mechanisms vouchers might do so.
State Sen. Alberta Darling's Recovery District Idea is Designed to Fail
Review of The Texas Economy and School Choice
In a recent report, The Texas Economy and School Choice, written by Arthur Laffer for the Texas Association of Business and the Texas Public Policy Foundation, Laffer evaluated the effect of the proposed Taxpayer Savings Grant Program (TSGP)—essentially a universal voucher program designed to provide school choice to every student in Texas. Laffer concludes that by raising graduation rates, improving education achievement, and thus increasing human capital, the TSGP would substantially raise wages and income for working families, thereby improving economic growth in Texas. In this review, we highlight two profoundly problematic areas of the report. First, Laffer’s assertions about the educational benefits of choice represent a severe overreach with and misapplication of the available research. Second, the author’s economic estimations are over-generalized and heavily biased towards those families who already have the wealth to choose and relocate. The report applies simplistic economic logic to education and fails to consider all but an extremely narrow and inappropriate slice of research on education, making the report unsuitable as a basis for public policy decisions.
Review of Federal School Finance Reform
This report argues that federal Title I funding needs to be reformed because, as currently structured, it does not address funding inequities between Title I and non-Title I schools, and because current regulations governing the program negate its effectiveness. The report proposes reforming Title I so that funding is portable, that is, it follows the child. That change would, among other things, facilitate school choice policies. To support its arguments, the report relies on two strategies: (1) it cites literature that supports its position while ignoring conflicting evidence, and (2) it uses rhetoric rather than evidence. The report ignores the complexity of Title I and of federal education funding generally. It also provides no analysis of the factors contributing to the alleged Title I problems. For these reasons and others, the report is little more than a polemic, using an eclectic assortment of disconnected facts and figures about Title I funding to promote choice and voucher policies. The report provides no evidence that its recommendations will improve academic outcomes and does not consider the adverse impact its recommended policies would have on improving educational opportunities, which is what Title I is designed to do.
NEPC Review: Pluck & Tenacity: How Five Private Schools in Ohio Have Adapted to Vouchers (Thomas B. Fordham Institute, January 2014)
The new Fordham report, Pluck & Tenacity, examines the impact of school vouchers on five private schools in Ohio. While the journalist who authored the report is primarily interested in the effect on this small set of schools, we focus here on an underlying assumption asserted in the executive summary of the report: that because of vouchers, “school outcomes will improve.” As presented in this report, this assumption about the beneficial impacts of vouchers is a case-study in how to engage in slanted selection and interpretation of research evidence. As we show in this review, the totality of three endnotes used in the report reflect not just an incomplete picture of the research literature on vouchers, but an extreme case of cherry-picking sources to support a contested policy agenda. Moreover, even with the few sources cited to put voucher outcomes in a favorable light, the report cherry-picks the findings that suit Fordham’s agenda, while ignoring the findings from those very same sources that do not support—and even contradict—the premise. Thus, the report is grounded in a twice-skewed and intellectually dishonest view of the research on vouchers and their academic outcomes. The subsequent journalistic celebration of five schools in Ohio then continues this unsystematic treatment of evidence, amounting to little more than cheerleading for vouchers.
Review of Report Card on American Education
The 18th edition of the American Legislative Exchange Council’s (ALEC) Report Card on American Education: Ranking State K-12 Performance, Progress, and Reform draws on ratings from market-oriented advocacy groups to grade states in areas such as support for charter schools, availability of vouchers, and permissiveness for homeschooling. The authors contend that these grades are based on “high quality” research demonstrating that the policies for which they award high grades will improve education for all students. This review finds that, contrary to these claims, ALEC’s grades draw selectively from these advocacy groups to make claims that are not supported in the wider, peer-reviewed literature. In fact, the research ALEC highlights is quite shoddy and is unsuitable for supporting its recommendations. The authors’ claims of “a growing body of research” lacks citations; their grading system contradicts the testing data that they report; and their data on alternative teacher research is simply wrong. Overall, ALEC’s Report Card is grounded less in research than in ideological tenets, as reflected in the high grades it assigns to states with unproven and even disproven market-based policies. The report’s purpose appears to be more about shifting control of education to private interests than in improving education.
NEPC Review: The Way of the Future: Education Savings Accounts for Every American Family (September 2012)
The Friedman Foundation recently published a report promoting Education Savings Accounts (ESAs). Like conventional vouchers, ESAs provide parents with public funds to purchase approved educational services, including private schools, online education, private tutors and higher education. The report presents ESAs as the optimal vehicle to bring Milton Friedman’s school voucher idea into the 21st century. While calling ESAs “the way of the future,” it lacks fundamental information to guide policymakers on their design, implementation, financing, and sustainability. These details ultimately determine the equity, efficiency and cost effectiveness of this proposal. For example, ESAs raise serious equity concerns: affluent parents could, if the policy allows, supplement their vouchers to purchase high quality educational services inaccessible to low-income families. This is indeed allowed in Arizona’s ESA program, the only existing ESA policy in the United States. Open to legal challenge, the report’s plan advocates using ESAs to sidestep prohibitions in state constitutions against supporting religious organizations with public funds. Unaddressed but relevant peer-reviewed evidence on school choice policies suggest that the claimed academic and economic benefits of ESAs are speculative and overstated. The absence of details and evidence in the report suggests it is ideological rhetoric rather than a workable policy proposal.
NEPC Review: The Effects of School Vouchers on College Enrollment: Experimental Evidence from New York City (August 2012)
This report examines college enrollment rates of students participating in an experimental New York School Choice Scholarships Foundation Program, which in the spring of 1997 offered 3-year scholarships worth up to $1,400 annually to low-income families. The study identifies no overall impacts of the voucher offer, but the authors report and emphasize large positive impacts for African American students, including increases in college attendance, full-time enrollment, and attendance at private, selective institutions of higher education. This strong focus on positive impacts for a single subgroup of students is not warranted. There are no statistically significant differences in the estimated impact for African-Americans as compared to other students; there is important but unmentioned measurement error in the dependent variables (college attendance outcomes) affecting the precision of those estimates and likely moving at least some of them out of the realm of statistical significance; the authors fail to demonstrate any estimated negative effects that could help explain the average null results; and there are previously existing differences between the African-American treatment and control groups on factors known to matter for college attendance (e.g., parental education). Contrary to the report’s claim, the evidence presented suggests that in this New York City program, school vouchers did not improve college enrollment rates among all students or even among a selected subgroup of students.
Review of Three Reports from the Comprehensive Longitudinal Evaluation of the Milwaukee Parental Choice Program
Milwaukee’s voucher program, the Milwaukee Parental Choice Program (MPCP), was created by state legislation in 1990 and enables low-income residents of the Milwaukee Public School district (MPS) to enroll at taxpayer expense in private schools that have been certified by the state Department of Public Instruction. The School Choice Demonstration Project at the University of Arkansas released in 2012 the final reports based on its five-year longitudinal growth study of this voucher program. The following reviews are of three of these reports: Report #29 compares voucher and MPS elementary and middle school test-scores; Report #30 compares graduation and post-secondary enrollment and persistence rates of MPS and MPCP students; and Report #32 compares test performance of MPCP and MPS students in grades 4, 8 and 10 in reading, math, and science. The reviewed reports use largely sound methods, but the data they assemble provide little to support the 22-year-old school voucher program. To some extent, this is because of specific methodological or analytical shortcomings. But it's also because the data and the reports simply fail to demonstrate that voucher schools are associated with improved outcomes.
Study's Results Are Flawed and Inconsequential
How Many Decades Before 'Reform' Becomes 'Status Quo'?
NEPC Review: The Comprehensive Longitudinal Evaluation of the Milwaukee Parental Choice Program: Summary of Fourth Year Reports (March 2011)
This review is of The Comprehensive Longitudinal Evaluation of the Milwaukee Parental Choice Program: Summary of Fourth Year Reports, published by the School Choice Demonstration Project, University of Arkansas. The report makes eight claims about the effectiveness of the program, most of them positive. On the key issue of achievement of students receiving vouchers, however, the report merely concludes that the program is not harmful. As the report’s title suggests, the evidence for all its claims is almost exclusively the researchers’ own work, with no reference to other academic literature. Importantly, none of their own referenced documents were peer-reviewed. Even as some of the report’s claims are in accord with the broader literature, their appearance in isolation makes for an overly simple evaluation of the MPCP.
Suggested Citation: Belfield, C.R. (2011). Review of “The Comprehensive Longitudinal Evaluation of the Milwaukee Parental Choice Program: Summary of Fourth Year Reports.” Boulder, CO: National Education Policy Center. Retrieved [date] from http://nepc.colorado.edu/thinktank/review-milwaukee-choice-4
NEPC Review: How School Choice Can Create Jobs for South Carolina (December 2009)
]The South Carolina Policy Council Education Foundation report, How School Choice Can Create Jobs for South Carolina, argues that school choice, in the form of vouchers to attend private schools, would create significant job opportunities in five poor, rural counties of South Carolina. The report, however, relies almost exclusively on results of an earlier study that has significant limitations in its methodology and execution, rendering its findings unreliable. The report also introduces questionable assumptions while extrapolating these findings to the five focus counties -- assumptions that drive the outcomes but are unlikely to hold in practice. As a result of its uncritical acceptance of an earlier flawed study and in its introduction of additional untenable assumptions, the report offers findings that are unlikely to be valid and is of little use in informing policymakers and the public about the effects of vouchers.
Suggested Citation: Roy, J. (2010). Review of “How School Choice Can Create Jobs for South Carolina.” Boulder and Tempe: Education and the Public Interest Center & Education Policy Research Unit. Retrieved [date] from http://epicpolicy.org/thinktank/Review-How-School-Choice