by Sarah Pollock, Katie Floyd, Cassie Davis, and Paul von Hippel
In April 2017, the Texas House of Representatives voted 103-44 to prohibit state funding of private schools. This ended any chance that the House would take up a private school choice bill passed by the Senate in March. The debate on private school choice was heated, but advocates and opponents mostly talked past each other. Texans did not get to hear a reasoned debate of the evidence on private school choice.
If you listen solely to the appeals of advocates, you get the impression that private school choice programs are almost invariably successful. A statistic often repeated by advocates is the claim that 14 out of 18 randomized experiments have found that government vouchers to attend private school raise test scores. This comes from a report authored by Greg Forster and published by the Friedman Foundation, an advocacy organization founded by Milton Friedman, who minted the idea of private school choice in 1955.
Opponents of school choice have rarely engaged this evidence directly. Instead, opponents argue that private school choice will “kill public education,” “siphon away critical funding,” and “prop up institutions with no public oversight or accountability.”
These are important concerns that merit debates of their own, but they do not challenge the claim that private school choice consistently improves academic achievement. If that claim were true, that how much would concerns about school finance and accountability really matter?
In fact, evidence on the achievement benefits of private school choice is not so clear-cut. The Friedman Foundation’s method of summarizing evidence has been criticized by researchers at the University of Arkansas (Shakeel, Anderson, and Wolf), who conducted a more rigorous meta-analysis of randomized experiments on private school choice, Although the Arkansas researchers were personally friendly to private school choice, their summary found the evidence was split:
- Across US studies, the average effect of private school choice on math and reading scores was indistinguishable from zero.
- Although the average effect was zero, the effect varied from place to place and time to time.
- An early voucher program in Milwaukee had a large positive effect, which was awkward for voucher opponents.
- A recent voucher program in Louisiana had an even larger negative effect—which came as a shock to voucher advocates. This was included in the meta-analysis.
- After the meta-analysis was published, a new randomized study came out showing that vouchers in Washington, DC actually reduced test scores (by about -0.1 standard deviations). That will not help the average.
- Effects abroad were similarly variable. Very large positive effects were achieved in Bogotá, Columbia, while effects in India were more muted.
The unpredictability of results means that we cannot know in advance how effective a private school choice program would be in Texas. Advocates can’t guarantee success, and opponents may not have the luxury of saying they told us so. Any school choice reform should be viewed as an experiment that needs careful implementation and rigorous evaluation.
This “forest plot” comes from the Arkansas’ researchers meta-analysis. For each experiment, the plot displays a point representing the best estimate of the effect, a confidence interval (whiskers) representing a range of uncertainty about the effect; and a box represents the weight given to the study, determined primarily by its sample size. Any confidence interval that crosses zero signals that the estimated effect is not statistically significant and the true effect could have been zero. The diamonds represent the average effect across all US studies, all foreign studies, and all studies together.
Throughout the debate about school choice, Texas lawmakers and policy advocates could have engaged the evidence in a way that informed citizens instead of muddying the waters and deepening political divides. At the end of the day, Texans were denied the chance to consider all of the evidence in a critical and balanced fashion. While it may be too late to recalibrate the conversation about school choice in this legislative session, the issue of school choice will return, and many other issues face our state on a daily basis. There are ongoing opportunities to raise the bar for evidence in policy discussions.
Texans can handle the truth.
The authors are at the LBJ School of Public Affairs at the University of Texas, Austin, where von Hippel is an Associate Professor, and Pollock, Floyd, and Davis are master’s students. We offer a more detailed summary of the Arkansas results is on our Wiki page.