Methodology: how we measure the accessibility of municipal websites

Version 1.1 · September 1, 2026 · Maksim Galatin & Claude (Anthropic) · licensed CC BY 4.0

Why a separate methodology was needed

Automated accessibility measurements are plentiful. Scanners such as axe-core produce them: they are cheap, reproducible, and they measure page markup.

Data on what happens when someone tries to walk the path by keyboard barely exists. A rule set inspects markup; it does not attempt to reach the goal. What is needed is a traversal that actually presses Tab on a live page and watches where focus lands.

We perform both measurements and compare them with each other. The gap between them is the main result of this work, and it only emerges when both checks run over the very same sample.

Part I. Automated measurement

What goes into the sample

For the United States — the official .gov domain registry maintained by CISA (cisagov/dotgov-data, CC0 licence, updated daily). From it we take organisations of type City and County: 11,659 records across 51 states and territories.

This is an important difference from a set assembled by searching. To the question "where did you get these towns from" there is an answer that can be verified in a minute: the entire official registry, not whatever we managed to find.

For Europe — municipal organisations from OpenStreetMap under the tags amenity=townhall, office=government and related ones.

How we check

A real browser (Chromium via Puppeteer), not an HTTP request. The reason: a large share of modern sites is assembled by scripts, and checking the source HTML measures something other than what a person sees.

On every site: we open the page and wait for the domcontentloaded event; we run axe-core — an open rule set aligned with WCAG 2.1 AA; we record every rule that fired, the number of occurrences and the DOM selector; we look for the payment page and check the path to it.

What we count and how

Violation density = the number of rules that fired divided by the number of pages checked. Density rather than an absolute count: a large site has more pages, and comparing by total violations means punishing size.

What we do NOT do — and this matters

We do not condemn an entire site on the basis of one page, we do not count warnings as violations, and we do not substitute judgement for measurement. The automated part answers one question only — did a rule fire — and no other.

Part II. Keyboard traversal

Here begins what no automated audit does.

The task set for the checker

Walk the path of a person who wants to pay a municipal tax or fine using the keyboard alone. The mouse is not to be touched at all.

Who does the checking

The traversal is performed by a software agent in a real Chrome browser, not by a person at a desk. We say this plainly, because how every number below should be read depends on the answer.

The agent drives a real page the way a person without a mouse does: it presses Tab, reads where focus has moved, and stops where there is nowhere left to move. It does not parse the source HTML and does not judge by markup — it walks the live page with all its scripts, consent banners and pop-ups.

What this gives us. Reproducibility: the same traversal can be run again and yield the same result, whereas a living person cannot be sat down in front of 67,000 pages and cannot be asked to repeat the path a month later. And scale: a measurement of this size simply is not done by human hands — which is precisely why such data did not exist until now.

What this does NOT give us, and we acknowledge it. The agent does not stand in for a screen-reader user and does not measure how comprehensible a page turned out to be. It answers one question — whether the goal can physically be reached by keyboard alone. We do not claim to have conducted testing with blind participants: we did not.

What is recorded for every site

The last field is the key one: it makes it possible to compare two measurements of the same object.

What counts as a barrier

• Focus is invisible (outline: none) — you cannot tell where you are
• Focus trap — the element cannot be left by keyboard
• Unlabelled field — a screen reader cannot say what to enter
• A button that is not a button — a div with a handler does not receive focus
• A modal dialog that does not close with Escape

How the traversal sample is chosen

Both groups are checked: the sites the automated pass judged accessible and the ones it rejected. Otherwise we would have a one-sided measurement showing only the errors that suit us.

Part III. What the comparison yields

The principal quantity of this research is the gap between machine and traversal, and it is measured in both directions:

False reassurance — the automated check says "accessible", the traversal does not get through.
False alarm — the automated check rejects the page, the traversal gets through.

State of the work as of September 1, 2026

Limitations we acknowledge

1. The US sweep is finished — the entire CISA registry has been walked, 11,659 sites out of 11,659. The US numbers are final; they will change only when other countries are added, and that will be stated plainly.

2. Not every record has a screenshot. Where the page did not open, there was nothing to capture.

3. A bot shield is not the same as inaccessibility. We do not know what is behind it and we do not record it as a barrier.

4. Traversal does not replace a live user. It answers one question: whether the goal can physically be reached by keyboard alone. The goal DIFFERS by page type, and there are eight: find a public document, file a 311 request, reach the payment step, find council minutes, reach main navigation, find a job posting, find a contact, find a dated event. The aggregate figures — 74.6 % did not reach the goal, 25.4 % did — cover all eight task goals, not any single one. The payment goal alone stands at 19.0 % reached.

5. The threshold of 40 Tab presses is ours. It is justified by the fact that the median successful path is three presses — but it remains our choice, not a standard. Tested by sensitivity analysis: thresholds of 40, 60 and 100 give IDENTICAL figures — nobody reaches a goal beyond forty presses, and the 99th percentile of successful paths is 35. Lower thresholds do change the result: at 30 the share reaching the goal drops by 0.6 points, at 20 by 2.1, at 10 by 5.7. The headline figure moves the same way: 53.8 % at 40, 55.0 % at 30, 57.6 % at 20. Forty is therefore headroom rather than a substantive modelling choice — any larger threshold yields the same answer.

6. Sites inaccessible to the automated pass used to drop out of the sample entirely. There are 787 of them, New York among them in full; 230 of those municipalities are bound by the 26 April 2027 deadline. We are covering them by traversal separately, in the same order — from large to small.

7. Some pages were checked twice, and we worked out what that changes. The traversal runs in rounds, and the same address occasionally comes round again: 348 records out of 95,524 — 0.4 % of the log. We recomputed every share keeping one record, the latest, for each address-and-page-type pair. The headline number did not move at all: 74.6 % before and after; the share reaching the goal, 25.4 % either way. A difference of one hundredth of a point, and we state it here so that it need not be explained later.

8. The observations are not independent, and we measured that. Eight tasks on one site are related: a badly built site tends to fail all eight. The intraclass correlation is 0.263 and the variance inflation factor 2.72 — eight tasks therefore yield not eight independent observations but roughly three. The effective sample size is 17,272 rather than 47,036 measurable records. The confidence interval for the headline figure, corrected for this: 52.7 % [51.7; 53.8]. Uncorrected it would read [53.2; 54.4] — almost twice as narrow, and it would be false. Separately: counting by municipality rather than by page — one vote per site — gives 52.7 % against 53.8 %. A gap of 1.1 points: the conclusion does not depend on the unit of count.

Openness

• Raw data is published in full, including records that do not suit us
• The checking code is open
• Traversal screenshots are attached — one for every page checked
• Errors in earlier versions are described in the methodology itself rather than rewritten after the fact

We checked ourselves too

A study about accessibility whose own sites are inaccessible is worth nothing. On 27 August 2026 we put all four of our own sites through the same procedure as the municipal ones: by keyboard, in a real browser, checking focus visibility at every step.

And here is what we found on ourselves. Our code contains 83 occurrences of outline: none — the very line that accounts for 31 % of all break-offs in our own research. They are harmless only because they are overridden by a global :focus-visible rule. That is exactly the difference between "written badly" and "works badly", and we demonstrate it on ourselves.

Conflict of interest disclosure

How to cite

Maksim Galatin & Claude (Anthropic). Municipal Website Accessibility: Automated Scan versus Keyboard Traversal. AIfa Works, 2026. CC BY 4.0. aifa.works/research/methodology

Feedback

Found an error in the methodology or the data — write to us: contact@codeofdigitaleternity.com

The data itself — aifa.works/research/data