Robots will be verifying some of our ballots. Can we trust them?

As a record number of voters cast their ballots by mail this year, more than half of the states are relying on signatures to verify voters’ identities. Apparently, mismatched signatures were the most common reason that mail ballots were rejected in 2016. And nearly 30 counties in at least eight states use automatic signature verification software to help — yes, software. While humans aren’t always that great, software makes mistakes too. Some experts worry it could contribute to voter disenfranchisement.

It’s a topic for Quality Assurance, where we take a deeper look at a big tech story. I spoke with Kyle Wiggers, who reports on artificial intelligence for Venture Beat.

Kyle Wiggers: Automatic signature verification software relies on algorithms to verify signatures, in this case, on ballots. What it’s doing is comparing signatures found on mail-in ballots with other signatures that come from state databases, usually motor vehicle databases. And what it’s looking at is certain features in those signatures, which could be the width of a signature and the height of a signature, and also what are called local signatures, which could be things like symmetry and stroke directions.

Sabri Ben-Achour: How often is it wrong, and how does that compare with how often humans are wrong?

Wiggers: Some studies show the accuracy could be as low as 74%. Other studies suggest it could be much higher — as high as 96%. But we don’t have benchmarks from the systems that are in use to verify signatures on these mail-in ballots. We basically have to go by what the manufacturers of the systems are telling us, which is that the systems are accurate. In some counties, the software has been adjusted to approve 40% of signatures or 20% of signatures. It’s really up to county officials to determine how high they want to set that threshold, which is problematic in many ways. But that’s the reality of the situation.

Ben-Achour: When I read your story, I immediately thought of my own signature, which is literally a line, like a line that a drunk person would draw. I mean, are there flaws in this software that’s supposed to read signatures?

Wiggers: Many of the experts I spoke to said that certain voters, especially those with mental or physical disabilities, or stress-related ailments or who don’t write in English as a first language are potentially at higher risk of having their ballots rejected. And when you’re dealing with algorithms like this, it’s always difficult to say how representative the datasets are that were used to train the algorithms. There’s also another point to consider, which is a lot of the signatures that are used to verify identities on these mail-in ballots are collected using electronic pads, and electronic pads are, of course, nothing like paper ballots. So needless to say, there are a lot of factors here that could be discriminatory.

Ben-Achour: Are human reviewers involved in the process at some point in the counties that use this technology?

Wiggers: Yes, correct. Some counties have a multistep approval process where software looks at these signatures first, and then humans and then there’s an audit afterward, but it varies depending on the state. Again, there are no consistent guidelines regarding this. It’s really kind of being determined, as we speak, what the best implementation practices are.

Ben-Achour: If your ballot is rejected, whether by a robot or a human, can you do anything about it? Do you get a second chance?

Wiggers: Yes. In 18 states, voters who experience a rejection of their ballot because of a missing signature discrepancy will be notified. That’s codified in law, but the protocols definitely vary by county. There was a study by the University of Florida that found that smaller counties often simply fail to mail notices. However, other counties are pretty good about it. In Sarasota County, for example, officials will send a letter to voters, and they’ll also try to alert them by text or call. And I’ve seen other counties that even go the social media route. They’ll try to reach out to voters via Facebook, if they have that information. But again it really depends on where you live. And the truth is that many voters probably might not be aware that their signatures were rejected or not verified properly, and may never get the opportunity then to correct that error.

Related links: More insight from Sabri Ben-Achour

Kyle Wiggers was telling us how robots may be imperfect at matching signatures, but it turns out people are really not so good themselves. Yes, there are trained experts used in court who can do forensic analysis with more than 90% accuracy, but that’s not who’s looking at ballot signatures. In some cases, it’s just spare staff from city hall brought in to help. And then you’ve got issues like how people’s signatures change over time. As people get older, their hands might move differently, and young people might be experimenting with different signatures.

People whose first language is not English may have trouble with consistent signatures, according to experts. And the number of ballots disqualified on the basis of signature has sometimes been larger than the margin of victory in a race. The LA Times has a really good write-up of this and some examples where you can see how well you do at identifying whether two signatures came from the same person. I did horribly.