The Pima County Election Integrity Blues

Orange_ribbon Please see the Election Integrity Homepage for complete coverage and the latest news.

The Democratic Party's Election Integrity Committee has been locking horns with Pima County over the security of the elections systems used in local elections. The result has been a protracted lawsuit that is trying to get access to public election records and answers to some fundamental security questions, a criminal investigation by the Attorney General's office into the actions of Pima County Division of Elections personnel, and a report issued by Chuck Huckelberry's office to the Board of Supervisors regarding election security and what his office is doing about it.

The issues in this dispute are somewhat detail-oriented, not terribly media-genic, and there are really a couple of areas of concern, not just one. The result of this sprawling and evolving set of facts is that the press doesn't do a very good job of informing the public about what the issues really are. Worse, the press' view that every story has two equally valid sides tends to allow some parties to the dispute to make wildly inaccurate and unsupported claims without the press really examining the facts underlying those claims. The result is that bullshit frequently doesn't get called on some folks who richly deserve to have their bullshit called.

I'm going to be taking a close look at some of the primary materials (the forensic report (PDF) the AG's office relied on, Huckelberry's report to the Pima Board, statements from the AG's office (PDF), and depositions from the pending lawsuit by the Democratic Party against Pima County (not online)) and try to distill the most important facts and issues so that you can understand what's really at stake in this controversy.

It's important to look at all this material together. If you only read Chuck Huckelberry's report to the Pima County Board of Supervisors on the election integrity controversy, for instance, you might get the impression that while there might have been some serious problems with election security in the past, there certainly wasn't any criminal behavior at all, and new security measures have put all such worries about the security of our elections to rest for good. Of course, that's exactly what CHuckelberry would have you believe.

GEMS and iBETA and Other Obscure Terms.

The heart of the matter is the security of the GEMS software that Pima County uses to tabulate votes. Election integrity activists noted discrepancies in the log files that were consistent with tampering with the vote and asked the Attorney General's office to take a look, since the Pima County Attorney's office had a conflict of interest because they are required to represent the County in the lawsuit.

During their investigation of these allegations of possible criminal tampering with the Regional Transportation Authority election, the Attorney General's office contracted with iBeta, a voting technology testing lab accredited by the U.S. Election Assistance Commission, for computer forensic services. Understanding iBeta's report is vital to gaging the appropriateness of subsequent government reactions, including Huckelberry's report.

I have some experience with computer forensics from my time at AOL with the Trojan horse and virus team, and from work defending clients against allegations of computer-based criminal activity. I don't claim to be an expert, but I can read a forensic analysis report. The key paragraphs of the iBeta report are astounding and deserve to be reproduced in full before commenting on them:

"During testing it was discovered that the GEMS software exhibits fundamental security flaws that make definitive validation of data impossible due to the ease of data and log manipulation from outside the GEMS software itself.

Ultimately, it is the determination of iBeta that the overwriting of the target file can be attributed to human error. iBeta arrives at the "human error" conclusion for two reasons:

1) iBeta was unable to detect any manipulation of the 051606 [editor - the RTA vote database] event data across the multiple copies of the data discovered.

2) The basis of the of the investigation is that there are log entries that point to tampering - but it is far easier to remove evidence of tampering from the logs that to actually tamper with the vote totals in the Microsoft Access database that the GEMS software uses. So it does not follow that someone with the knowledge to manipulate the GEMS data would neglect to alter the log file to remove the evidence of the manipulation."

You probably want to read that a few times. There are some startling statements and some interesting premises that we need to examine in those paragraphs.

"The GEMS software exhibits fundamental security flaws that make definitive validation of data impossible..." What this means is the GEMS software is not capable of providing any assurance that the data files have not been altered. There is no way to be sure that the data hasn't been changed. I'd say that providing such data security is a pretty fundamental function of voting software. GEMS can't provide even the most rudimentary level of security.

Unfortunately, that software has some quite serious security issues; in fact, it would fair to say that the software essentially has no security. Jim March, a board member of election integrity advocacy group Black Box Voting, demonstrates the software's profound vulnerability in this video:

The GEMS software was not designed for elections; it is general data capture and tabulation software that was retrofitted and marketed for this purpose. Because it wasn't built with security in mind, it is, quite simply, a security nightmare. It's fine software for many non-secure data compilation applications, but it is a miserable joke to use in when security and auditability is a primary concern.

You can essentially boot up MS Access and edit the data tables to alter all the voting data (the actual vote counts) and access logs (records of who accessed what and when) and leave no trace of your tampering. The GEMS software is fundamentally flawed and cannot be made secure, yet, as we shall see, Pima County continues to use it, and has no plans to stop using it, and is instead relying on a flurry of security procedure chaff that attempts to cover up the simple fact that the software they are using is worse than useless.

The conclusion that there was no tampering is essentially meaningless given that "definitive validation of data [is] impossible." This fundamental contradiction becomes apparent when iBeta gives the reasons for their conclusion. iBeta did not detect any manipulation, but that is to be expected, because there would be no evidence if someone tampered with the data and knew what they were doing. But because iBeta did see irregularities in the log, which would be easier to manipulate than the voting data itself, the irregularity must be due to human error, not tampering.

Did you get that? The data wasn't tampered with because a competent tamperer would not have left any evidence. There was some evidence, thus there must not have been any tampering. This is not computer forensics, it is amateur criminal profiling.

Is there any evidence of tampering? There are irregularities in the logs which could be consistent with rudimentary data tampering by overwriting files with new tampered data files. But rather than admit that there is no evidence that excludes the possibility of tampering (largely because the GEMS software is incapable to providing any degree of data security) iBeta concludes, based only on the fact that the evidence in the logs was not eliminated, that the discrepancies are due to user error. And if there were no evidence of tampering would they have concluded that the election data had been tampered with? Of course not.

So, how does the Attorney General characterize this damning report that concludes that GEMS is fundamentally flawed, and that evidence in the log files consistent with tampering is actually user error because a real tamperer wouldn't leave any evidence? "Attorney General Terry Goddard today announced that his office did not find evidence that anyone tampered with a Regional Transportation Authority election in Pima County last year." Really? I thought if there wasn't any evidence that would be evidence of tampering according to iBeta. I kid. But, the statement is clearly far too categorical. It isn't false, but it isn't accurate either.

The release goes on to say, "We did not find any evidence that the computer technician at the center of this case manipulated this election. However, the consultant's report reviewing the system did raise serious concerns about election security." Well, I suppose it's getting more accurate, but Goddard still hides behind the failure to find any positive evidence when, in fact, there is evidence of irregularities, but iBeta decided that it represented only user error and not an attempt to manipulate the vote. Goddard's statement also fails to mention just how fundamentally serious the security concerns are, or how they might be addressed. The most effective thing would be to immediately stop using fundamentally flawed software! Granted, Goddard's report does indicate that their "review did reveal election security weaknesses that need to be addressed by Pima County." Very diplomatic, but hardly it hardly speaks to point of assuring secure elections.

When it comes to the bottom line, however, the AG gets it dead wrong. The AG's statement claims that iBeta's testing "determined that no data was changed in this election." That is misleading to the point of simply being false. iBeta concluded that they could not determine if any voting data was changed because the software is fundamentally flawed. Not being able to tell is not the same thing as determining no data was changed. iBeta's ultimate, and in my view, specious judgment that the log discrepancies were due to user error has no bearing on whether any voting data was changed. iBeta's only relevant conclusion was that they wouldn't be able to tell if vote data was changed.

Another bone of contention in the Democratic Party's pending lawsuit against Pima County is a demand for disclosure of the actual voting data to the plaintiffs so that they can do their own forensic analysis; they've only had access to the logs, in which they also noted the same discrepancies that iBeta dismisses as user error. So far, the Democratic Party has had to take Pima County's word on anything to do with the actual data. The reason given is that state law requires governments to keep the program code of electronic voting vendors secret: cozy deal, huh?

Pima County claims that the data files with the voting data are part of the GEMS program and thus cannot be released. That's like claiming that your term paper is part of your word processing software. It's completely absurd. Huckelberry's report touches on this peculiar claim saying, "In fact, it is precisely because of our concerns for election security that we have opposed the release of electronic information databases on the grounds that such information is not only made confidential by law, but also because such release would make future elections more vulnerable to attack."

You know, CHuckelberry has a point here. If I was using software that had fundamental security flaws, I would want to keep it under wraps, too. Unfortunately, secrecy is one of the worst forms of security there is. It would serve the voters of Pima County, and the security of future elections, much better to just use software that wasn't fundamentally flawed. Then you wouldn't have to twist yourself in semantic knots trying to avoid releasing your data, because its release wouldn't make future elections vulnerable to attack.

Who's Got The Summary Reports?

Another issue at the heart of the case against Pima County Elections Division is that employees frequently ran summary reports which revealed the progress of races before the close of voting. Such a report, if accessed prior to the end of an election, would essentially be a hyper-accurate poll which could be used to influence the outcome of an election. Creating such a report is not a crime by itself, but sharing that report with anyone with the purpose of influencing an election would be.

Once a hard copy of such a summary report is created, it can become very difficult to account for it. Such reports could end up anywhere if not promptly destroyed. Was there evidence that such reports were systematically destroyed according to an established policy? No. Yet the AG's office nonetheless concluded that "Mr. Crane did not share with anyone the results from these summary reports," and concluded their investigation.

The Attorney General's investigation did not speak to relevant witnesses like Robert Evans, supervisor of the Elections Division warehouse, who helped run the equipment that generated these summary reports, even though the lead investigator was told he would give relevant testimony. Under oath during his deposition by the Democratic Party, Mr. Evans had lots of interesting things to say about those summary reports, all of which directly contradicts what was told to the Attorney General's investigators by the subjects of the inquiry.

The first thing we learn from Mr. Evans is that Bryan Crane, the IT supervisor who was the focus of the AG's inquiry, was not the only one to run summary reports, or to have access to them. Mitch Etter and Brad Nelson and Mary Martison (employees of the Pima County Elections Division) and Kathy Cuvelier (an Oro Valley city clerk employee) had all printed, caused to be printed, or had been given access to summary reports, according to the deposition of Mr. Evans. Did the AG's investigation consider all these folks? They only mention Bryan Crane.

Further, Mr. Evans testified that Mitch Etter and Brad Nelson would regularly remove those reports from the computer room in which they were generated, and he did not know what was done with them. Where did those reports end up? Were they destroyed? Shared with others? There are strong suggestions in Mr. Evan's testimony that those reports were, in fact, shared with interested parties, possibly the Oro Valley clerk's office or town council. Who else might have been interested in getting summary reports? Given that the printing of these reports, their removal from the counting room, and their possible sharing with outsiders was a regular occurrence over a number of years, the AG reports' acceptance that there reports were only used to double check results seems rather too trusting.

It doesn't seem that the AG's investigation gave any credence to information provided them by the Democratic Party and its counsel, or that those leads were even investigated. I don't wish to think that the investigation consisted merely of a cursory interviews with the principals in the Elections Division sufficient to create a report that accepts their testimony as gospel truth, avoiding any embarrassment for Pima County, while avoiding collection of any information from witnesses that might require further inquiry. I would like to know what happened to all those summary reports and whether they ended up in hands they weren't supposed to, but the AG's deferential investigation and its cherry-picked conclusions leave me far from convinced that what happened here was anything other than a convenient whitewash.

So What's the Big Deal?

The report that Pima County Administrator Chuck Huckelberry's office generated in response to this whole kerfuffle is a masterpiece of misdirection. It is laudable that the Elections Division has started looking seriously at the security of our elections, but that is their job, after all - one might wonder why it took a lawsuit by one of our political parties to get them off the dime. But the impressive litany of security measures that Huckelberry's report lays out is missing one very important bullet point: the GEMS software itself.

They are going to all sorts of extremes in electronically isolating equipment, enhancing access controls, installing video surveillance, upgrading their procedures to ensure that printed reports are dealt with appropriately and that observers have appropriate roles, and instituting new training. And Pima County is apparently willing to spend as much as $10 million on doing all this. That is all to the good, and, I might add, is all due to the pressure brought to bear on an arrogant and unresponsive bureaucracy by a tenacious pro bono trial attorney, Bill Risner, and a cadre of committed election integrity activists in the Democratic and Libertarian parties. All the voters in Pima County owe them a big thank you.

But the GEMS software itself, as iBeta's report makes so clear, is fundamentally flawed. It provides no data security at all. You can wrap a flawed system in as many layers of security as you like, but it remains a flawed system that is simply unsuitable for the conduct of elections. Huckelberry's report seems to recognize the problem: "Furthermore, iBeta stated that its testing revealed that the GEMS software exhibited "fundamental security flaws that make definitive validation of data impossible due to the ease of data and log manipulation." It is this finding that further strengthens our commitment to ongoing improved physical security as well as implementation of more checks and balances."

So close! It's right there in front of them: they need to address the software issue. But they instead draw the conclusion that they need 'physical security' and 'checks and balances'? Really? Wouldn't a real commitment to security include giving voters the assurance that their vote data could be definitively validated, so that a bozo with a copy of MS Access couldn't steal their democratic voice?

Instead they are clarifying admin rights on their computers, instituting dual passwords and ballot chain-of-custody procedures, enhancing records retention, and even running hash tests to validate the integrity of the GEMS software. We would be better off if they would simply trash GEMS and use software that is not fundamentally flawed. Why won't Brad Nelson and Chuck Huckelberry do that, when common sense so clearly points to this being the most crucial and most obvious security improvement?

I don't know.

Even speculation finds little purchase on that enigmatic nut. But if I had a way that I could secretly change the outcome of any election at my whim, with absolutely no way to detect that tampering because the software was fundamentally flawed, I have to wonder if I would want to give that up. I would hope that I would, but human nature being what it is, I don't think I could categorically say. I might build the walls around that system so high and secure that people could feel a little better about the gaping security black hole at thr center of my elections system, but I might not really want to give up the ability to nudge a number here or there.

Now, I wouldn't for a moment suggest that anyone in the Pima County government has such an improper and unworthy motive in continuing to insist on using the GEMS software, but I also couldn't blame anyone for arriving at this uncharitable conclusion, either.

11 Comments

Wow! Excellent work, Mike. Do you have any idea what software is used in other parts of the state?

Posted by: Zelph | November 12, 2007 at 10:26 AM

Unfortunately, MS Access is even worse than your analysis. Essentially (and without getting too geeky) Access does not really implement the concept of transactions that are at the heart of all major database systems. The reason transactions are important is as follows: if you have write a vote for a candidate and then write to the log file and perhaps another operation, Access has no way of determining or ensuring that they happen in order or that all of them happen. In other database systems (SQL Server, Oracle, Sybase, Informix etc), when you create a transaction, the system will not allow you to partially complete a transaction. This means that if you have a three step process and any of the steps fail the system rollsback the entire process and returns the data to its previous state. It would then presumably generate some sort of error message.

Furthermore, security in Access is nearly non-existent. Anyone that does a Google search for hacking MS Access passwords will quickly find several programs that will allow you to crack the passwords in Access.

Finally, MS Access is notoriously corruptible. I have personally seen half a dozen corrupt access databases over the years, but none of them were doing anything as important as counting vote totals. It is not outside the realm of possibility to think that you could corrupt individual databases based on precinct and influence an election. If a database on a machine or in a precinct became corrupt they only way to fix would be to rerun all of the ballots.

At the end of the day, why would we allow a database to be used for tabulating votes that most small companies do not consider relievable enough for any mission critical system. I worked for a company that was forced to throw away their MS Access order system by an auditor because it was not considered auditable and open to fraud.

It is not even a matter of cost. Most large database systems have fairly inexpensive distributable versions of their databases. It is unconscionable that we would certify a voting system that we would not use for any auditable commercial activity to tabulate our votes.

Great blog entry…

Posted by: ademlament | November 12, 2007 at 11:28 AM

This is the best summation of what is a very complicated subject that I have seen yet. Thank you, Mike. By the way, while Bill Risner is doing a great deal of this work pro bono, there are a lot of expenses associated with these depositions and other legal details that have been paid by the Pima County Democrat Party. If any of your readers are inclined to support this work financially, I am sure the Party would appreciate contributions.

Posted by: Laura | November 12, 2007 at 12:05 PM

Zelph, I don't know what other Arizona counties are using, but that would be an excellent thing for readers to find out and leave a comment on...

Posted by: mbryanaz | November 12, 2007 at 01:13 PM

Three points:

* Bill Risner isn't working pro-bono.

* The "dual passwords" thing is a disaster. What they're really doing is "binary passwords", where a single password to get to *everything* election related (for a single username called "admin") is split between two people. In other words, the initial login has to involve two people. But after that, you can't track which human being did what. This is security 101: individual logins let to track personal accountability. Chuckleberry and company are dead set against having our election system track personal accountability of users. Why? Because Diebold made it a pain in the keister to switch users while heavy processing is going on: you have to shut down all the data feeds, close GEMS completely and re-start it under the other username, then restart the data flow. Tough. They're going to have to.

* On the video I did, it will look better at higher resolution - there's a button in there to have it jump to the Google Video page and run at pretty close to the original 800x600 resolution, at least on most people's screens. In the embedded video the text isn't readable.

A final comedy note: the Diebold "GEMS" program has a Windows icon like everything else. The icon? A fist holding a globe, I kid you not. It's a more colorful version of the corporate logo for "Dr. Evil" in the Austin Powers movies...we should be checking Chuckleberry's office for the presence of midgets and bald cats.

Jim March

Posted by: Jim March | November 12, 2007 at 11:19 PM

Jim,

Unless someone has an ace card up their sleeve my
gut feeling that the pending/ongoing RTA vote
case(s) may just fade onto the dust pile.

With my small back gound in the computer/software
and election processes makes me say I agree with the
Pima democrats election integrity concerns.

The RTA vote saga brings forth an important possible
"lessons learned" idea.

The "lesson learn" idea is:

1. Implement a no-nonsense post election voting
machine manual hand count of ballots as follows:
a) Count ALL positions on the ballot.
b) Determin voter intent and record same.
c) If the manual count does not agree with the
machine tabulation, the manual count shall be
used as the "official" count for that machine.
d) 100% of all voting machine should be verified.
e) Or, a sample number of machines should be
verified somewhere from 2% to 12%.
1. The error margin shall be one tenth of the
"recount rate". the error margin would then
be 0.01%
2. If the error between the machine tabulation
and hand count is greater than 0.01% at any
position on the ballot perform a hand count
of a second sample lot.
3. If the second sample lot fails the 0.01%
perform a hand count of the remaining
machines.
f) The post election sample test/inspection/count
should use a standard such as MIL-STD-105 as
a guide and information in the mechanism of
"why use sample testing...etc."

2. 20/20 vision is great after any event. But,
if, a post election verifiication was in place
at the May 06 RTA vote, and, the RTA vote result
had a changed outcome the Pima citizen would have
the upper hand in asking and demanding the Pima
official of what went wrong and what is the fix.

When we use voting machines for any election at the
national, state, or local level it appears that the
Pima democrats are on the right tract, in that they
want to make sure the following steps are queaky
clean and open to the public:
a) federal level approval/certification of machines
b) state level approval/certification of machines
c) pre-election ballot programming of machines
d) pre-election test and seal of machines
e) election day use of machines
f) end of election day machine tabulation print
g) end of election day close and seal machine
h) POST ELECTION VOTING MACHINE VERIFICATION
i) etc.

Feel free to contact me,

Thanks,
Frank Henry
Tel: 928-649-0249
e-mail: fmhenry4@netzero.com

Posted by: Frank Henry | November 13, 2007 at 12:09 PM

Okay. Just for the record should this post end up in the hands of a judge determining legal fees in the future: I used pro bono as short-hand that people can easily understand, not as a legal characterization of the actual client agreement between Bill and the Democratic Party. Bill Risner is not working pro bono, but he's not being compensated by the Party right now, and possibly not ever. It is a deferred compensation arrangement, of a sort. The point being, he's certainly not doing it for the fee, and he's extending his personal resources and time to pursue the matter.

Posted by: mbryanaz | November 13, 2007 at 12:43 PM

Jim, et al,