What the scanner cannot see

The axe step in CI is green on the review gate and has been for months. That is worth something. The question this lesson exists to answer is exactly how much, because at ninety-odd portfolio companies the pipeline is the only mechanism that scales, and a standard that overestimates its own automated coverage is not a standard — it is a green checkmark with a company behind it.

Both of the standard tools tell you their own limits, in writing, in places nobody reads. This lesson is mostly a matter of quoting them accurately and then doing the arithmetic on what remains.

axe is a rules engine with a deliberately low ceiling

axe-core is the open-source engine underneath almost everything: the browser extensions, the Playwright and Vitest integrations, and, as it happens, Lighthouse. Its own README states the coverage claim plainly — “With axe-core, you can find on average 57% of WCAG issues automatically” — and it sits alongside the design commitment that produces that ceiling: “It returns zero false positives (bugs notwithstanding).”

Read those two together, because they are the same fact stated twice. Deque — a vendor that sells accessibility tooling, audits and training, so treat the 57% as a vendor’s figure about its own product rather than an independent measurement — has chosen never to report something it cannot be certain about. A rule ships only when it can be decided mechanically. Everything ambiguous either becomes an incomplete result flagged for human review, or gets no rule at all. The 57% is not a technology limitation waiting on better software; it is the direct consequence of a correctness guarantee, and no amount of engineering removes it.

Two details about how the ruleset is organised that change what “axe passes” means:

• Rules are tagged, and not all of them run. The rule documentation groups rules by tags including wcag2a, wcag2aa, wcag21a, wcag21aa, wcag22aa, best-practice and section508. The WCAG 2.2, AAA, experimental and deprecated groups are disabled by default. A green run against default configuration has not evaluated the 2.2 criteria at all, which means it silently agrees with whatever version your conformance note happens to target — or silently disagrees, and you will not be told which.
• Rules are not criteria. There are roughly 105 rules across every category, against the whole Level A and AA set in WCAG 2.1. The mapping is many-to-many and lopsided: some criteria have several rules, and some have none at all. Counting passing rules tells you nothing about criteria covered.

The most instructive gap, given where this course is going: the rule documentation contains no rule at all for aria-live, role="status", or live regions generally. 4.1.3 Status Messages — the criterion the confidence chip lives under, and the criterion an entire streaming interface hangs from — is essentially outside automated reach. Not badly covered. Not covered.

Lighthouse is axe with a score bolted on, and the score is the risk

Lighthouse’s accessibility category is “a weighted average of all accessibility audits,” with “weighting based on axe user impact assessments”, and each audit is binary: partial passes earn nothing. So the engine underneath is the same engine, inheriting the same ceiling, with an additional layer of judgement on top — Deque’s impact ratings, converted into weights, converted into a number out of 100.

Google states the limitation itself, in the string Lighthouse prints above the accessibility section of every report it generates:

a11yCategoryDescription: 'These checks highlight opportunities to
improve the accessibility of your web app. Automatic detection can only
detect a subset of issues and does not guarantee the accessibility of your
web app, so manual testing is also encouraged.'

That is from Lighthouse’s own default configuration — Google documenting the limits of its own product, which is the strongest kind of vendor statement there is, because it cuts against the vendor’s interest. Print it out. When somebody arrives with a Lighthouse screenshot, that sentence is already in the screenshot, directly above the number they are pointing at.

The score is the specific danger, and it is a danger for a reason established in the conformance lesson: AA is not a percentage. Conformance at AA means satisfying every Level A and every Level AA criterion, with no partial credit and no arithmetic. A Lighthouse accessibility score of 96 is a weighted average of a subset of checkable rules. It is not 96% of AA, it does not round up to AA, and it is not evidence of conformance at any level. It is, however, a number, and numbers travel upward through an organisation in a way that caveats do not. If you standardise on Lighthouse for portfolio reporting, you have chosen to publish a metric that will be misinterpreted as a conformance level roughly every time it is quoted.

What lives in the gap

The W3C is the one party in this conversation with nothing to sell, and it is blunt: “Web accessibility evaluation tools can not determine accessibility, they can only assist in doing so”, and “Tools cannot check all accessibility aspects automatically. Human judgement is required.” It also warns in both directions — “Sometimes evaluation tools can produce false or misleading results” — and closes with the sentence worth carrying into every tooling decision: “avoid relying too much on what tools say over addressing the real-life experience of website users.”

Made concrete against the criteria from the previous lesson, the gap has a recognisable shape. In every case the machine can check that something is present and cannot check that it is right:

• Alt text. A rule confirms the attribute exists. Whether alt="image" describes the source crop of an extracted address field is not a machine question.
• Link and button names. A rule confirms an accessible name exists. Whether “View”, repeated eleven times down a table, tells anyone what they are viewing is not.
• Focus order, 2.4.3, Level A. The DOM order is fully machine-readable. The criterion is about whether the order preserves meaning and operability, and meaning is not a property of the DOM.
• Error suggestion, 3.3.3, Level AA. A rule can find the error text. Whether the suggestion it makes would help a person fix the field is a reading-comprehension task.
• Status messages, 4.1.3, Level AA. No rule exists. And even if one did, appropriateness — is this worth interrupting for, is it announced once or forty times — is nowhere near the reach of static analysis.
• Accessibility supported. The judgement WCAG explicitly delegated to you cannot be delegated onward to a scanner, because deciding whether a pattern is supported requires hearing it announced.

There is one more limit that the coverage debate tends to obscure, and it comes from the conformance requirements rather than from the tools: conformance is for full pages and complete processes. A scanner runs against a rendered page in a state. Every state it did not reach — the error state, the empty state, the mid-run state, the state after a dialog opens — is unexamined, and in an agentic surface most of the interesting states are ones a crawler never gets into.

What this does not cover

This module ends with a conformance note that has a scope, real findings with levels and costs, and an honest account of who checked what. What it does not have is any evidence about the half the scanner cannot see, because acquiring that evidence means building the patterns correctly and then listening to them.

That is the next module, and it starts from the fact underneath every 4.1.2 failure: ARIA changes what a control is announced as and changes nothing else — no keyboard behaviour, no focus management, no styling. The role-is-a-promise lesson is where that gets stated properly, the four-patterns lesson works through combobox, disclosure, dialog and tree against the Authoring Practices Guide, and the screen-reader lesson is where the accessibility-supported judgement left open in the conformance lesson finally gets something to stand on.

Then the part none of this tooling reaches at all. axe has no rule for live regions, and the reason is not neglect — it is that nobody, including the W3C, has written down what the right answer looks like when the live region is attached to a token stream. The announcing-the-stream lesson opens on that gap rather than pretending it is closed.