The following is a letter to Colorado SOS staffer Dwight Shellman and members of Colorado’s Pilot Election Review Committee:
I'm including a message I sent to Dwight long ago. Please include this in published public comment to the PERC committee on the SOS website. ( http://www.sos.state.co.us/pubs/elections/VotingSystems/reviewCommittee.html ). Please send me the plans/schedule for the mock RLA and the test data dumps you got from the vendors. And please send the spreadsheet that contains the schedule of election activities. I am shocked that it was not published on the SOS website, as far as I can tell. I have certificates of appointment for a statewide ballot issue committee to watch in any county. While watching last week I also discovered by chance that the SOS issued apparently secret provisional watcher rules to some counties and asked them to pilot these. Fortunately the four pages of rules did not catch me by great surprise as they were the product of a committee I served on. Why wouldn't the public be made aware of that watcher pilot? Perhaps not your department. As you know I am attempting to evaluate the tabulation accuracy of the four systems as well as their facility to support transparency in the service of election verification- not only in the future via a RLA (Risk Limiting Audit) but the facilitation of deeper dives into verification of election results for accuracy. At least one such deeper dive is definitely appropriate to do prior to this UVS (Uniform Voting System-s) decision. I am attempting to do it and am facing ridiculous costs on the order of $20,000 - a sum accumulated from estimates of some of the counties for access to ballot scans. This has forced me to constrain my requests with the hope of getting useful access to evidence from these elections in a much smaller scope. My efforts will be something closer to a pilot of the functional part of an "evidence based election" - something that election quality advocates like myself and many election officials seek. I have no doubt that the counties have been corresponding about how to answer my requests. the CORA requests are rendered challenging by the limitations in CORA law and the opportunities of new technology at hand. Perhaps Adams and Denver have been responding without much coordination. Most other counties seem to think it will take 5 seconds to find identifying marks on one side of a ballot scan shown on a computer. I think that would be excruciatingly slow and disastrously expensive. If the SOS office has exposure to these conversations by email I would appreciate seeing the content of the discussions. I know you were cc:d one one county response to my request. It is important for the process of public access to election records to be a part of the legislative agenda and rules agenda for the future. Huge opportunities can be missed by failing to address the benefits of some or all of these new voting systems. One such benefit would be for campaigns to locate potentially mis-adjudicated ballots and to systematically include any such discovery in the process of auditing the election so that higher accuracy could be obtained prior to certification - particularly with a narrow margin contest. For my part I have posted (at considerable expense of my time) all of the threads of my conversations with the counties regarding the CORA requests for ballot designs as well as scans of tabulated ballots, cast vote records, and new scans of duplicated ballots. These conversations represent a large range of facilitation of public access to verify elections. No doubt we will not achieve a "uniform election system" by enforcing the use of one or two or whatever prescribed voting systems. The UVS is a well distributed, well worn myth and the counties seem to know it. I hope the SOS does. I hope he becomes well aware of this transparency project and the benefits that it might produce. The record of these CORA conversations under my commentary is located here: https://electionquality.com/2015/11/colorado-pilots-of-new-voting-systems-allow-technical-evaluation-through-access-to-ballots/ Please be aware that the running of a normal low volume election with new voting systems does not necessarily produce the evidence of how these systems will perform in exceptional conditions or in case of narrow margins. We must simulate these conditions or dig deeper into the evidence to see how they are performing in detail. One such exceptional condition is the inevitable presence of people like me who will attempt to verify the accuracy of an election. I have not noticed much simulation of that condition in the counties where I have been watching. In most cases I am the only credentialed watcher who has appeared. And of course thanks for making this a public process- something that some other states do not feel necessary. Simply the opportunity to be involved is a very meaningful enhancement that I have every reason to appreciate and compliment. And thanks to the tireless efforts of many who are engaged in trying to get Colorado to a better election. Harvie Branscomb -------- Forwarded Message -------- Mon, 21 Sep 2015 14:42:40 -0600 Dwight Here I reprise my email of July 29 that refers again to the SOS preparation for RLA. I understand that the past two PERC meetings were cancelled and that includes the meeting that would have included Philip Stark. On 7/29 I asked again if I should ask you once more for the communications regarding RLA with the pilot counties. I do not even yet know which 4 counties will conduct the RLA. I am heavily involved with the Election Verification Network in a discussion of RLA and have been for years. Philip Stark and Ron Rivest and many election officials and academics are part of that conversation. It is clear that Colorado's actions in November will be of great importance to the audit community nationwide and I plan to learn as much as I can about what Colorado is doing as well as to learn as much from the pilots. I plan to CORA the PDFs of the ballot styles from the 8 counties involved in the audit so that by the time I get ballot images I may be in a position to independently tabulate them
-------- Forwarded Message --------
Subject: |
are we ready for risk limiting audits? |
|---|---|
Date: |
Wed, 29 Jul 2015 16:18:38 -0600 |
From: |
Harvie Branscomb |
To: |
|
CC: |
Dwight Shellman Here is an excerpt of something I am sharing with election verification experts:
Fortunately as Phil Stark pointed out ballot level risk limiting audits do not require aggregating multiple ballots and only actually require capture and comparison of patterns off of single ballots at a time. However this is also not trivial and even newest systems are not necessarily providing data that has good human factors for this purpose.
It appears that machines are better than humans at counting up the number of pieces of paper in front of them. It seems unavoidable that machines fail to capture idiosyncratic voter intent when humans succeed. Therefore an ideal combination is found in the sort and stack method that can be combined with machine counting of pages. For example during a single contest recount it would be ideal to have human judges separate the ballots by voter choice and batch tabulate them such that all ballots in a batch (or on a particular scanner or memory card) are hand sorted to be a single voter choice. The scanner can then provide a machine count of sorted pages and a sub-tally of vote counts to confirm that it recognized all as voted for the specific choice. This will easily expose discrepancies that can be resolved by human observation and improve recount accuracy.
I am currently analyzing the first 8,000 of 130,000 ballot images and cast vote records from the Dominion system used in Denver CO (the much advertised and visited May municipal pilot election). The principal reason for my analysis is to learn what we need to know to guide the evaluation of such a pilot system from the point of view of auditing (and transparency)… auditing the audit as it were.
The format for the Cast Vote Records that were provided 1) as a separate text file aggregated by batch as well as 2) the “AuditMark” voter selection digital printout that is embedded with the 200dpi 1bit per pixel ballot image are not suitable for convenient machine tabulation and they aren’t particularly desirable for hand comparison either. Human usability concerns (as well as machine formatting compatibility concerns) abound in the design of forms and scanned images and reports that are needed for auditing. No doubt this issue is not limited to Dominion.
Our CO pilot review program seems insufficiently concerned about the ways that the risk limiting audit will take place. Perhaps more attention will be paid within the next few weeks. We have until November when five such pilots will take place as part of the 2015 election. At present it is uncertain what the instructions for a risk limiting audit will be and it is even more uncertain how we will know if we have done a good pilot audit.
Dwight I am deeply concerned that our pilot evaluations will fail to measure and document the crucial back office aspects of the systems even as they are in use in real elections. The evaluations of these systems in real elections crucially must look beyond the motions of successfully performing an unremarkable election. The RLA is a big part of my concern and that is why I asked you to forward the communication to the vendors about preparations for the pilots. Should I ask for that again? I am hoping that I will get enough information from Denver's completed pilot in time to be able to make a sample partial evaluation of that system as an example for what the PERC or SOS could and should do as a minimum to evaluate each system. I am hoping I can do that before the PERC decisions on how and what to evaluate are made. No doubt what I am doing by myself is not enough either. I am already learning that the RLA may be painful with some systems. And the counties performing the pilots may have the tendency to act to defend the system rather than subject it to a critical evaluation in public. That should be a major concern of PERC and SOS and all Coloradans. I don't think my bi-weekly writings to the PERC have been productive but I do think that you can benefit from the information I am collecting. By random chance after examining only a few ballot images in Denver I already found an incorrectly captured voter mark in the Dominion May pilot- but the CO evaluation must be in a position to find all of them for each piloted system. And system evaluators must be in a position to understand and evaluate the remedies offered by the vendors. A technique such as I have briefly described above for sorting before scanning if used for a top of ticket contest would get part of the way there and without spending much extra time. Likewise the documentary products of each system (logs, settings, etc.) must be studied in detail and each document's ability to convey exception cases and discrepancies evaluated. Unfortunately I have not yet received any logs or reports from the Denver system, and the ballot images are coming slowly due to an understandable learning curve on privacy redaction. The timing of the PERC and any other SOS evaluation must allow for weeks of evaluation of reports and cast vote records and audits after the documents are produced. It will not be enough to say no one complained. And on that matter, it is also important that well trained watchers/observers be present at the 10 pilots at important moments throughout - of course including the ballot design, LAT and ballot prep and early tabulation and post election activities. And plans must be made for a super-LAT at each pilot that plans for potentially problematic ballots to be voted and results analyzed. Remember what the clerks did to ballots for the UVS demo last year- crumpled them, etc. Whatever the out of state certification is doing and whatever SOS is doing in house is probably insufficient to provide confidence about exceptional test cases and the ability of systems to detect, log and remedy these cases. Finally you keep mentioning the hardware test and acceptance tests. These are important and have been ignored in the past. I think work should be done on what the hardware tests (every election) and acceptance tests (by county upon purchase) should be- and these tests should be performed with careful evaluation during the pilot or within its scope. Seening the lights come on and the system say "ready" doesn't provide enough information. Please send this to the PERC if you feel it is constructive to do so. PS I just received the following link from Ron Rivest- another very well respected statistician working on elections - with a simple one page description of a RLA. Unfortunately it is still only the mathematicians description and not one for election officials. Someone should turn this (or similar mathematical instructions) into a clear and specific multi-county process (all 5 RLA pilots coordinated plus any other counties separately doing RLA?) for the state-wide issue on the ballot and within county audits for the remaining contests. http://people.csail.mit.edu/rivest/pubs.html#Riv15y In this list of publications, the RLA description is number 311 or the second from the top at the moment. Harvie Branscomb