Fool's Gold: the Goldwater Institute Civics Test

by David Safier
I posted yesterday about Matthew Ladner's poll showing that high school students don't know basic civics. I'm sure he's right, and I think schools should do better, but I gently accused him of playing "Ain't it awful?" with the data -- taking a pot shot at public schools by hyping a genuine finding.

Today, no more Mister Nice Guy. The survey is crap, and Ladner's reporting of it is crappier. It's typical of the worst kind of G.I. "research," where you take your information, look for the results you want and publish those as if they are gold plated findings.

Let's look at a few quotes from Tuesday's G.I. daily email, authored by Ladner.

The report "reveals only 3.5 percent of Arizona high school students have learned the basic history, government and geography necessary to pass the U.S. Citizenship test."

Note the precise figure: 3.5 percent.

"We hired a firm to interview 1,140 Arizona high school students and ask 10 questions drawn at random from the exam given to applicants for United States citizenship."

So, over a thousand high school students were surveyed, and only 3.5 percent of them could pass a U.S. Citizenship test.

This morning, the Star wrote about the survey in its editorial. It was a telephone survey. That means the students who took the test had to be home and willing to take the survey. They sat at home, out of school, probably no more than half interested in what they were being asked, and answered questions they had no stake in answering thoughtfully or correctly.

How seriously did they take their answers? When asked who was the first president, some answered, George Bush, Bill Clinton and Barack Obama. Someone who has never met a teenager might think answers like that show the depth of their ignorance. Anyone who had spent five minutes with a teenager in the last year would know those are flippant, "I don't give a damn about your survey" answers. Matthew, no one -- no one -- believes Barack Obama is the first president of the U.S. Those students are putting you on.

What was the most popular answer to all the questions, according to the Star? "I don't know." That answer is a typical dodge. Any teacher knows the teenager is really saying, "I'm bored with this, and I don't want to think about it, and I'm not sure I know the right answer so I won't even try because I might embarrass myself. So get out of my face. Are we through yet?"

How different would the results have been if the students were chosen randomly, sat down in a room with pencil and paper and given some kind of incentive for answering correctly? I'm betting the scores would have been significantly higher,and nearly everyone would have put down "Washington" as the first president.

In the email, Ladner didn't mention it was a non-random phone survey. But he was perfectly happy to use a percentage-with-a-decimal-point number for the passage rate: 3.5%. That's simply intellectually dishonest. Not incorrect. Intellectually dishonest.

This is the kind of thing that infurates me about G.I. They have oodles of money, they do lots of research and studies, but when they come up with a result they like, no matter how questionable, they put it out there like it's nobel prize material. They never seem to ask themselves the kinds of questions any honest researcher or scholar asks: "Can I really use a precise figure like 3.5 percent on a non random survey conducted over the phone with disinterested teenagers? No, I'd better reveal my methodology and make my figures more approximate."

I began my Fool's Gold series when Ladner took his favorite $9,500 per student figure for Arizona, using expenses no one else includes, and compared it against other state's education spending figures, then concluded we were in the middle of the pack in per student educational funding. The assertion was so ridiculous, he eventually had to take it back. But he and G.I. continue to use the "shock and awe" method of releasing research findings. Take the most shocking numbers you can cook up, put them out there and stick with them.

You're basically a serious guy, Matthew, and as smart as you need to be. If you want to be thought of as something other than a propagandist, take what you write more seriously.

Comments

[flounder]

Like I commented on your other post about this, I would have never heard of Ladner if he didn't go around the state lying about Arizona's 49th in education ranking.
This makes Ladner, above all other things, a liar.
There are plenty of things that Arizonan students could learn more about, but sending out a liar to tell us what those things are doesn't really work.

[cranky]

LOL They SERIOUSLY did a PHONE survey with TEENAGERS?!?! OF COURSE the results are inaccurate! There they sat, playing XBOX and their phone rings with a call NOT from the friend(s) but some "researcher" who totally wants to harsh their mellow by asking stuff about history(by the way, if they were under the age of 18, did they have their parents' permission to participate?).

Here's what I'd like to know, did these telephone researchers call their home phone (see above question re: parental permission) OR did they call their cell phones? If this research team called these kids on their cell phones, I'd be really concerned. Let's for get for a moment all of those "rollover minute" commercials where teenagers are reprimanded for throwing away "money". More alarming is how that information would be collected. How would they identify whether or not a teenager would answer the cell phone? Information -- private information -- would have to be disclosed at some level that could constitute a violation of privacy.

But forget these points. They just fall into the bucket of things that bother me about the manner in which the GI and/or Mr. Ladner gather/report their "research": second hand and at a distance.

The bigger peeve in this is it shows the lack of interest the GI takes in children and kids. It appears as though no attempt was made to hire a research firm that knew and understood kids/teenagers. Otherwise the setting/medium in which this study was conducted would have started in an entirely different way. Like in front of an XBOX.

Send your post to the Star, Mr. Safier.

[Matthew Ladner]

David-

Both you and the Star are off base on this. It has long been established in survey research that the vast majority of people DO NOT want to admit ignorance on a survey. For example, I and most other poor souls who went through a graduate political science program learn survey research by examining the National Election Survey. They've been giving these since the early 1960s.

The survey includes literally scores and scores of questions of detailed policy and political matters, and the vast majority of people will sit there and opine away in a multiple choice format. "Do you support the SALT II Treaty?" "Do you support the public option health care proposal?" etc. etc. etc. Don't know/no opinion is always an option, but very very few people take the option.

Now, on the other hand, in an open question format, in any given year only 40 some-odd percent of the American public can correctly identify the Vice President of the United States. It's been that way for decades.

You either know that George Washington was the first President of the United States, or you don't. Everything we know about survey research indicates that the vast majority of people hate to admit ignorance.

Further, we employed a reputable survey firm that employed proper polling techniques. Trust me cranky, these guys are well aware of possible sources of sample bias (like cell phones) and do everything they can to get a good sample.

Furthermore, you are also off base to present this as some sort of drive by shooting of public schools. We also paid money to have private school students surveyed and reported the results, which are simultaneously much better and still pathetic. There has been no effort to put lipstick on that pig.

[todd]

Was the survey if the private school students done on the phone or at school?

[Ben Kalafut]

I sort of hate to say it, but you're spot-on about intellectual standards at the Goldwater Institute--and the merits of this survey--here. I know that Ladner and some of the other staff up there are capable of good research, but they have no reservations about feeding us crap so long as it's covered with a good layer of icing. It's as though the Patrick Michaels tactics on climate change have been adopted by the GI as standard practice.

Making things worse, they play up their credentials. "Dr Art Laffer", "Byron Schlomach PhD", etc. The reports are a somewhat subtle insult to intelligence, but these "Dr" and "PhD" things are an outright statement that they think we're stupid and easily dazzled.

Send me an e-mail backchannell (bennett at kalafut dot us) and I'll forward you some hilarious stuff from Byron Schlomach. Most people get more serious and cautious when they're caught screwing up. Not at the Goldwater Institute!

OT: Your 'blog has become full of cruft and takes forever to load. This is the first time in days that I've been able to get the "Post a comment" box to pop up; it took several minutes on a high speed connection at the university.

[flounder]

Notice how Ladner didn't deny he is lying about the funding rankings, even though he whined about everything else. How does lying about the issues make America more civic minded?
I don't think it does, in fact, I think such lies are extremely contrary to building civic education and only fuel apathy towards all political debate and knowledge.
Running the liars like Ladner out of our discourse and generating hostility and even mockery, at all levels of debate, toward the lies like Ladner spreads would be the very first step I would take towards improving interest in our American political system and history.

[Matthew Ladner]

Todd-

Yes both surveys were done over the phone.

Ben and Flounder-

Surely you can do better than hollow ad hominem attacks. Flounder's last paragraph sounds to be right out of the fascist handbook, which is a good match for an unwillingness and/or inability to accurately describe debates that have been held here.

Finally David, if I were actually a propagandist, why survey private school students and report the disappointing results? It would have been easy to do. I could have presented the horrible public school results and called for greatly expanded school choice.

[Ben Kalafut]

Matthew Ladner:

When someone claims his intelligence is insulted, speaks of Patrick Michaels tactics, and infers that intellectual standards are low, it's not an ad hom attack. It's ellipsis.

The interesting thing here is that (unlike this anonymous coward "Flounder") I'm largely on your side concerning education policy. Our common cause would not be well served by my not acknowledging when you are throwing bad argument after good, nor in my not noting that intellectual standards seem to be slipping a bit.

Human instinct is, unfortunately, largely on "Blue Meanie"'s side; liberalism, especially markets, is a counterintuitive and uniquely civilized way of thinking. Since we're up against this it's important to be above the opposition, to be better than the next guy, to be correct and modest and to stick to strong arguments, even if the other side can get away with playground rhetoric.

[todd]

Matthew,
While you did test private school students, neither that fact nor their poor results were mentioned in the press release. Only after reading the full report was that clear. The link to full report no longer seems functioning on the Goldwater site, so even this bit of information would be rather difficult to come across.

I think it is quite clear the context in which this report was released - that of an epic budget battle being waged largely about funding for public education. This report certainly insinuates that this abysmal result is from a failure of public education, as opposed to perhaps wider causes.

Looking at charter and private schools could provide important perspective on this problem, however, there are some problems with these figures.

For charter schools the number of students surveyed was 136 and this is justified because this is proportional to the number of regular public high school students. Unfortunately, this justification makes no statistical sense. According to the AZ Dept. of Ed, there are ~35,000 charter high school students. In order to get the decent sample size (95% confidence level and +-5% confidence interval) one would need to survey 380 students. As it is, the 136 gives a +-8.39 confidence level for each question.

The ADE says there is 276,000 regular public high school students, making the confidence interval of that survey around +-3%.

Essentially this means that the differences shown between charter and regular public school students is statistically meaningless and so are claims comparing charter and regular public schools such as twice the number of charter school students "passed" (6/10 or more correct answers) versus regular public schools.

For the private school numbers there are clear differences as the private school portion has a large sample (not sure total private high school enrollment in the state but the sample size used would provide quite accurate measures).

Here are the percentages for private and public schools in terms
of number of correct answers.

Private Public
0 1.2% 2.3%
1 8.5% 12.9%
2 17.6% 26%
3 22.4% 29.2%
4 21% 17.5%
5 15.6% 8.6%
6 9.6% 2.7%
7 3.6% .8%
8 .5% 0
9 .01% 0
10 0 0

So what does this tell us? 4% of private schools students got 7/10 correct vs. .8% for public schools. 13.7% of private school students "passed" vs. 3.5% regular school students. One can't deny the private school students performed slightly better even considering any confidence interval, but still is the conclusion we should draw that that there is something uniquely wrong with just the public schools?

If the Goldwater Institute wants to make this comparison meaningful, it might be helpful to remove from the private school sample students who are attending private schools with admissions tests. Then we would actually have something to compare. As it stands right now, I see no evidence that charter schools are doing anything better than regular public schools and one could easily draw the conclusion that the best way for schools to raise its scores on these tests is to have entrance exams which filtered out underachievers. Perhaps then maybe we can actually have a real discussion about why knowledge in these areas is so poor.

[flounder]

Why is pointing to your old routine of lying about education funding considered an "ad hominem" attack? An ad hominem attack would be me saying you are ugly.
And there is nothing fascistic about a desire to see (paid) liars treated with the respect they deserve, which is little.

[Ben Kalafut]

Getting OT: at least we know who Ladner is. There are hundreds of "todd"s in the state and probably no talking flatfish. It's one thing to call someone a liar anonymously, it is quite the other to stake one's reputation on one's statements.

[Tucson Vice]

It sounds to me that Mr. Ladner is (in part) quoting from the book "Just How Stupid Are We?" which itself relies heavily on the same election surveys that he mentions.

If I am wrong and he hasn't read this book, he should. It demonstrates through decades upon decades of these surveys that the level of civic knowledge remains essentially the same, regardless of most outside factors. It has nothing to do with the state of today's schools (which can only be blamed on republican budget cuts and teacher firings) and everything to do with the fact that most young people (and adults) are not nearly as interested in US government and civic understanding as they are in other things (work, hobbies, family etc).

[flounder]

I use the name flounder rather consistently on the internet. You are free to Google 'flounder', read my comments, and critique me in the same fashion as I have for Ladner. If Ladner was lying under the pseudonym 'creepo' I would still charge 'creepo' with being a liar, based on evidence, not on what the byline said.
Further, I've read the DHS report on right-wing terrorists, and I remember when Michelle Malkin learned some people's names (whose positions she did not like yet couldn't refute) and had her minions send them death threats. I stand on my words; the only reason you would want to know my real name is for the "ad hominem" attacks you claim to detest or because you want to 'Malkin' me.

[todd]

" There are hundreds of "todd"s in the state and probably no talking flatfish"

Mr Kalafut - I try to argue based on analysis and data and it is not clear why you are choosing to take issue with me because of my login name. I would also point out that this type of public advocacy is Mr. Ladner's career and profession - it is not mine. If you want to argue with what I am saying please do, to argue against me because of some irrelevant fact is the classic definition of "ad hominem."

[Ben Kalafut]

"Flounder": Clearly you missed my point. If you think I'm going to sic the friendly live-and-let-live capitalist readers of Goldwater State on you, you maybe need to take whatever meds they're giving the paranoid types these days. And guess what: That ain't libel, because there's no such person as "flounder". My point is that you can say whatever you like without having it "stick" or follow you. You can exaggerate Ladner's errors to "lies" and it doesn't follow you.

"Todd": I used to agree with you. Then I discovered the Internet, the way anonymity poisons the waters, and that playground tactics can and do work to undermine reputations, fairly or not. My beef about your anonymity (and others') is that you can say anything at all without it affecting your reputation and without it coming back to bite you socially.

In this context it's a remark contrasting your conduct and Ladner's, and not on the merits of your arguments, so no, it's not a classic "ad hom". Ladner may treat us like we're stupid, but at the very least he stands by his statements. "Todd", on the other hand, does not. Nobody has time to verify everything; trust is important. Between "todd" and "Matthew Ladner", to whom am I capable of giving any trust?

I suspect this is also why this "Blue Meanie" character can foam at the mouth in ways that Safier or Bryan can't. Meanie isn't a person. He's a character. Safier and Bryan are people.

[Matthew Ladner]

Flounder-

I've explained this to you in the past, but I'll do it again. In the discussion about public school spending, I backed up my claims with links to state documents like this one:

http://www.ade.state.az.us/AnnualReport/AnnualReport2008/Vol1.pdf

and this one:

http://www.azleg.gov/jlbc/allfunding.pdf

Whether you simply can't read these reports or else lack the emotional maturity to update your opinion when presented with empirical evidence doesn't really matter. The fact that you would write...

Running the liars like Ladner out of our discourse and generating hostility and even mockery, at all levels of debate, toward the lies like Ladner spreads would be the very first step I would take towards improving interest in our American political system and history.

...simply reveals you to be illiberal know-nothing. If you have any evidence of my having lied, feel free to produce it. Until then the person you are making a mockery of is only yourself.

Todd-

I've written three pieces since the publication of the study (two blog posts and one as yet published blog posts) and the bad private school results have been included in all of them. I agree with you that the charter school results have a larger margin of error, but the point of this has never been to describe how well charter or private students have done. I pretty sure I've used the term "pathetic" to describe them.

[todd]

Mr. Kalafut,
I've been on the internet since the days when the only forum for discussion was usenet and have sometimes used my full name and sometime not and other times using handles as that was whatever system being used enforced. This has been the norm on the internet for a long time and is part of the online culture. I have no reputation as a public figure and my views would carry no more or less weight if people knew who I was or what I do for a living. You are certainly free to think or say what you want, but it does seem odd to burst into a forum in which first names or handles are part of the norm and start criticizing people for using them instead of engaging with the actual arguments being presented.

[todd]

Mr. Kalafut, One more thing, your comment is an attack on me and not my arguments. It is classic ad hominem, which you accuse others of.

[flounder]

Even an extreme right-winger like Superintendent Tom Horne thinks you are a liar:
http://www.azcentral.com/arizonarepublic/local/articles/2009/04/12/20090412edbudgets0412.html
excerpt:
[The libertarian-leaning Goldwater Institute includes capital expenses such as the cost of building new schools. With that approach, Arizona spends $8,528 per student. Calculating expenditures should include all sources of revenue, said Matthew Ladner, the institute's vice president of research.
Horne disagreed, saying that per pupil expenditures should calculate only money spent teachers and programs in the classroom.
"Construction cost is a result of the fact that we're a growing state," Horne said. "And that's a product of growth. It's not an effort that we're making in education."]

[David Safier]

I've heard Horne say the same in a television interview. And a similar statement precedes national per pupil estimates. They specifically say construction costs are kept out of the comparison since they are a function of population growth, not of educating students.