by David Safier
Friend of the blog Matthew Ladner has done it again with not one, but two studies based on surveys using methodology so poor, no self respecting statistician would take the results seriously. And his conclusions? Let's just say, even if they were accurate, they're pretty underwhelming.
Ladner's two studies for the Goldwater Institute are titled, Better Citizens at a Lower Cost: Comparing Scholarship Tax Credit High School Students to Public School Students and Tough Crowd: Arizona High School Students Evaluate Their Schools. The links take you to summaries of studies maintaining that private school students are getting better educations because they are more politically tolerant, are more willing to volunteer and like their schools better than public school kids. Pretty weak tea if you ask me, even if they were valid conclusions based on solid evidence, which they're not. From the summaries, you can link to the studies themselves.
The Goldwater Institute folks are in a bit of a bind right now. They've seen one of their favorite government programs, tuition tax credits, take a well deserved beating in the press and know it will take a further beating from a bipartisan legislative group set up to investigate both the credits and the School Tuition Organizations administering them. G.I. can't defend the blatant abuses of the program -- in fact, it has worked to position itself as part of the move to reform the system -- but it desperately wants to create the impression that the program is worth saving and even expanding to a full blown voucher system, which is G.I.'s ultimate goal. And it wants to get the information out into the media --quick! -- to defend tax credits against their detractors.
So G.I. is desperate to show private schools are better than public schools to justify the program. Because if private schools are better, the more students who attend private schools using tax credits and/or vouchers, the better educated our children will be.
But there's one big problem. All the recent studies comparing traditional public, charter and private schools indicate that none of them is superior to the others academically.
Bush's DOE funded a report that explored test scores at various schools and concluded, if you compare similar students, those in private schools perform no better on standardized tests than students at traditional public or charter schools. The only exception is, students at conservative Christian private schools score lower than everyone else.
Studies in Florida comparing voucher students with public school students showed no appreciable difference in test scores. The experiment in D.C. where a large group of students attended private schools on vouchers provided the closest thing to a controlled experiment we've ever had, and it also showed no significant difference in scores. And that's comparing public schools in D.C., which has one of the most dysfunctional school districts in the nation, to private schools. That's not the result voucher proponents were hoping for.
The only place they've been able to find a difference is in attitude. Students and their parents appear to like their private schools better than equivalent students and parents in public schools. That's all. Conservatives would make fun of liberals if they used touchy-feely information like that instead of hard test score data, but that's all they've got, so the conservatives are trying to make the most of it.
Ladner sets out to show that Arizona students in private schools like their schools better and have more positive attitudes toward tolerance and diversity than their public school counterparts based on a random survey of public and private school students. Since G.I. is an advocacy group masquerading as a think tank, the burden of proof that its study is objectively accurate is very high. But Ladner's study doesn't come anywhere near the necessary burden of proof a university would require of an objective researcher, or even a student writing a term paper.
WONK ALERT: The rest of this post, after the jump, is a dry-as-dust discussion of the problems with Ladner's methodology. Unless you're interested in that kind of thing, skip it. I hate to bore people. But it's important to show what a shoddy, shameless piece of propaganda G.I. has come up with, and that takes time.
If you want to show that private schools help students have better attitudes than public schools, you have to compare the attitudes of similar students in the two types of schools. Obviously, you can't compare rich students at a private prep school with low income students at a poorly performing public school and get results that mean anything. If Ladner's studies can't show that the two groups of students surveyed are similar in every way but the one important variable -- attending a private or a public school -- his results are worthless.
Ladner realizes this, so he tries his best to show he has two equivalent groups, and fails miserably.
I'll begin with Better Citizens, Lower Cost: Comparing Scholarship Tax Credit Students to Public School Students. The study wants to show that private schools create students who are more politically tolerant and more willing to volunteer their time than public school students.
The study is based on a random survey by Strategic Vision of 1,350 high school students in Arizona who were either attending public high schools or attending private high schools using tuition tax credits. Ladner spends a few pages trying to demonstrate that the two groups come from similar income groups. He begins with an admission that the two groups might not actually be equivalent.
Proxies for family income that we do have available for public and tax credit students, however, imply less imbalance than might be suspected.
"Less imbalance than might be suspected." It's not nearly as bad as you might think, he says. He's on shaky ground already. So he says, OK, to show you how similar the two groups are, let's look at the number of students on free lunch at public schools and compare that with students getting tuition tax credits.
In the 2006-07 school year, 40.7 percent of Arizona public school students qualified for a free- or reduced lunch under these federal guidelines. During that year, a family of four could earn a maximum of $37,000 per year to qualify for the program.
Fine, let's look at the three STOs that means test their scholarships (many others are happy to give scholarships to children of millionaires) and see how honest Ladner is being with us.
Ladner knows Arizona School Choice Trust well, since he sits on its board. So it shows a lack of candor that he doesn't state the STO gives scholarships to children from a family of four that makes as much as $57,260. That's over $20,000 more than free lunch students in public schools. Already, we're skewed toward a more affluent population.
And what about the Catholic Tuition Organization of the Diocese of Phoenix? Yes, it means-tests. But here's what it says on its FAQ page:
Are there any income limits when applying for aid?
There are no income limits when applying for assistance. Each family’s unique financial situation is considered when assessing need.
I couldn't find similar information about the Tucson Diocese's STO, but it's clear that even the STOs Ladner picks out to make his point actually refute it. Scholarship students from the STOs that means test can be from higher economic bracket than free and reduced lunch students.
Ladner wants to go even further to show how rigorous he is, so he digs himself in even deeper.
In other words, the poorest of the poor don't show up much in private schools, but since quite a few of them drop out of public schools, it's pretty much the same thing. Even Ladner must have blushed when he wrote that paragraph.
A study is only as reliable as its weakest link, and the economic link here is shaky indeed. I would also say a study is no more reliable than its author. Ladner shows he is perfectly willing to distort his argument to make his case stronger, which puts his reliability, and his conclusions, in doubt.
To strive for a higher standard of reliability myself, I'll admit that one factor leans in Ladner's favor. The gap between the answers of the private and public school students is pretty large, averaging about 17%. That would make up for some of the economic disparity between the two groups. But if the groups were more equivalent, that gap would very likely shrink to the point where it was marginally significant at best, and very possibly statistically insignificant. But we'll never know, because the methodology is so shaky.
Now, to the other study, Tough Crowd: Arizona High School Students Evaluate Their Schools. This study relies on a survey by the same company, also of 1,350 high school students. I'm going to give Ladner the benefit of the doubt and assume this is a different survey from the one used for the previous study, even though both have exactly 1,350 students. If not, there's a serious problem, because this time Ladner says the survey includes all private school students, not just those receiving tuition tax credits.
For this study, Ladner doesn't even try to create an equivalence between the two groups.
The demographics of the public school sample broadly match those of the public K-12 population. This report details the results of the students’ school evaluations.
An item by item examination of the data, however, reveals no significant differences between students attending private schools with the assistance of a tax credit scholarship (43 percent) and those not receiving such aid.
No rigor here at all. No attempt to say tuition tax credit students are equivalent to public school students, only that their answers are similar to those of other private school students.
The entire purpose of this study is to say that private school students like their schools better than public school students. Ladner doesn't mention that, even if this is true, the positive attitude doesn't translate into higher standardized test scores, which makes the importance of the finding questionable, especially to someone like Ladner who values nothing so much as test scores.
But here's the problem beyond the poor statistical sampling or the irrelevance of the finding. Private school students have a psychological motivation to say they like their schools. Otherwise, why would they and their parents have put out the extra effort to seek out the school and spent money for tuition and supplies that would have been unnecessary at a public school (most tuition credit students still have to pay some tuition, not to mention other expenses)? Here's a rough analogy. One teenager chooses to drink Coke and pays for it with his own money. Another is given a Pepsi whenever he/she asks for a soda. Chances are the Coke drinker is going to say, "Damn right Coke is the best drink out there!" while the Pepsi drinker says, "Yeah, it's pretty good, I guess. I've had better." In fact, neither soft drink is objectively better than the other.
If there's any doubt left Ladner has no respect for facts that get in the way of his conclusions, the statement he makes in both studies about how tuition tax credits save taxpayers money seals the deal. Here's what he says in the Tough Crowd study:
In 2007, Arizona school tuition organizations (STOs) gathered more than $54 million in contributions from the original scholarship tax credit. In that same year, 27,153 students used these scholarships to attend private schools. The scholarships averaged $1,788, much less than the $9,700 per student per year the state spends to educate the same student.
Ladner once again pulls out his $9,700 figure, discredited by liberal and conservative scholars and by Tom Horne (Horne has said more than once, the figure is ridiculous). But beyond that, Ladner arrives at his $1,788 figure by assuming all the students at private schools getting tax credit money would otherwise be at public schools. In fact, every study shows that a significant number of tax credit students, possibly a large majority, would be in private schools with or without the credits, meaning that every tax credit dollar they receive is an added burden on taxpayers. Both of Ladner's figures are based on dumbing down the data to the point where it is meaningless.
Ladner draws his questionable conclusions from weak survey data. Here's my more reliable conclusion based on my reading of his studies. When you arrive at marginally significant conclusions based on questionable data, as Ladner has done, you don't end up with much of anything -- except maybe a package you hope to sell to unsuspecting members of the media to get your message out.
Mr. Safier,
Three things you seemed to have overlooked...
1) The DC and Florida programs are saving the taxpayers money.
2) These programs are likely raising the performance of those public schools 'competing' for these vouchers.
3) Parent and student satisfaction is significantly higher for those using the vouchers.
If students are performing similarly on standardized tests, aren't these reasons enough to implement vouchers?
If you really want to understand how vouchers can affect an individual’s life watch the videos from the parents and students themselves here…
http://www.voicesofschoolchoice.org/
Posted by: AZ Ed Watch | September 11, 2009 at 08:47 AM