The error that made our numbers look better: how we found it and removed it

In short

We measure website accessibility: whether a person who does not use a mouse can move through a page with the keyboard and get what they came for. Over the past week we crawled tens of thousands of commercial websites in the US.

On 8 September we found two errors in our own crawl. Both made our numbers look better than they were. We removed them. Here is what they looked like from the inside — because this is exactly the kind of thing that other people's reports tend not to show.

The first error: an obstacle page counted as a flawless website

Large chains defend themselves against automated requests. Our crawler would arrive at such a site and receive not the site but a check page: "Robot or human?"

Here is what happened next. The server replied with status code 200 — "all good". The page rendered. The accessibility engine ran against it and found zero violations. And that is true: a page with one line of text and one checkbox has nothing to violate.

The log entry read: measured, no violations.

In other words, a website that refused to let us in was being counted as exemplary. There were more than four thousand such records, 3,227 of them with a round zero.

The second error: one chain weighing as much as a thousand companies

We take website addresses from an open registry of organisations. For chain businesses, every location is a separate record, and each one carries the same address in its "website" field.

The measurement: 57,401 checks performed across 8,170 unique domains. Seven checks per domain. One well-known coffee brand was measured 5,739 times — once per coffee shop.

If you compute the share of "sites with violations" by the number of checks, a single chain gets the weight of five thousand independent companies. And large chains usually have good websites: entire departments work on them.

What changed in the headline number

share of sites with violations by checks (wrong) ..................... 60.0 % by unique sites, captchas removed ..... 77.9 % shift ................................. 17.8 percentage points

Note the direction. We were understating our own result. Fixing the error produced a number that sounds more favourable to us.

That is precisely why we are writing about it. A correction that makes your result worse gets made out of caution. A correction that makes it better is easy not to notice and not to look for — and that is the most dangerous property of both errors: they raise no suspicion. The data looks fine. Nothing crashes. Nobody complains.

How we found it

We were not looking for either of them.

We were checking something else entirely: that every measured site had its own screenshot rather than a substituted one. To do that we counted the share of unique screenshot checksums. It came out at 74.4 % — meaning a quarter of the screenshots repeated.

Then we opened the repeats and looked at them. In the very first three examples there was a large retail chain's site with the title "Robot or human?" and zero violations.

From this comes a simple conclusion we are taking into our method: a screenshot checksum is a cheap and very strong check. It catches two different classes of error at once — a substituted measurement and duplicate input data. It will now be in every crawl we run.

What we did NOT do

We did not stop the running crawl.

The check showed that what needs fixing is the counting, not the collection: every log record preserves the page title, the domain, the status code and the screenshot checksum. That is enough to separate obstacle pages and collapse chains after the crawl. The data is intact; nothing needs to be gathered again.

This is the second time our number went down, not in our favour

On 1 September we counted 1,498 dead domains among municipal websites. On 8 September we recounted: 1,441. Fifty-five domains we had marked dead answered on retry, and two more on the third attempt.

The larger number looked more impressive. We kept the smaller one.

A checksum you can verify out loud: 1,441 + 10,461 = 11,902 — every municipal site in our sample.

What we checked separately

We ran the same procedure against our municipal crawl — the one whose data is already published openly. There is no skew there: 95,524 checks across 11,902 unique sites, eight pages each, zero obstacle pages. Municipal sites do not use captchas.

The published data needs no correction. It is only needed for the commercial crawl, which is still running.

Why we are writing about this at all

Because a number without a denominator is not a measurement, it is an impression. When someone says "78 % of websites are inaccessible", the only honest question is: 78 % of what exactly, and what did you drop from the denominator?

We answer in advance: the share is computed over unique sites, not over the number of checks, and obstacle pages are not in the denominator. Without this correction we would have had 60 %; with it, 78 %. We took the number we can defend.

The data, the code and the screenshots are open. Check us.