Accessibility Testing
Automated rules catch a real but limited fraction of accessibility defects — and the ones they cannot decide are the ones that stop people using the product. Keyboard and screen-reader passes are not optional.
The intent, the obvious build, and why it breaks
Every lesson starts where the work starts: someone wanted an outcome, and the first implementation that comes to mind has a problem.
What can an automated accessibility check actually decide, and what must a person still do by hand?
A person who navigates with a keyboard, a screen reader, voice control or magnification wants to complete the same task as everyone else — not a reduced version of it, and not with a support call.
Add the accessibility rule checker to CI and fail the build on violations. When it reports zero, the page is accessible.
A rule engine decides only what is decidable from the tree. It can prove that a control has no accessible name; it cannot decide whether the name "Click here" is useful, or whether it matches the visible label a voice-control user will say.
- A rule engine decides only what is decidable from the tree. It can prove that a control has no accessible name; it cannot decide whether the name "Click here" is useful, or whether it matches the visible label a voice-control user will say.
- Focus order is a property of a journey, not of a snapshot. A checker looking at one rendered state cannot see that opening a dialog leaves focus behind it, or that closing one drops focus to the top of the document (Focus Management).
- It cannot decide whether the flow is *operable*: whether every step can be completed without a pointer, whether a custom control responds to the keys its role implies, whether a drag interaction has any alternative (Keyboard Operability).
- Announcement quality is invisible to it. A live region can exist, contain text, and still announce nothing useful — or announce it three times, or interrupt what the user was reading (Live Regions and Announcement).
- Rules can be satisfied in ways that make things worse:
aria-labelvalues added to silence a violation, roles applied to make a checker happy,aria-hiddenused to remove something from the tree rather than fix it (The Rules of ARIA). - The result is a page with zero reported violations that a screen-reader user cannot get through, which is the specific failure mode this lesson exists to prevent.
What is actually happening
In the browser, not in the framework.
- An automated check walks the accessibility tree the browser computed from the DOM and evaluates rules against nodes: is there a name, is the role valid here, do these attributes contradict each other, does this contrast ratio meet the requirement (The Accessibility Tree).
- Those rules are conservative on purpose. A rule that produced false positives would be turned off, so engines only assert what they can decide from the tree — which is why the coverage ceiling is structural rather than a gap in any particular tool.
- Published analyses of automated coverage put it at roughly a third of the accessibility defects real audits find. The exact fraction varies by tool, ruleset and page, and it is a useful order of magnitude rather than a constant.
- The remaining defects need a judgement about meaning (is this name accurate), sequence (does focus go somewhere sensible next) or experience (is this announcement comprehensible) — none of which is derivable from a single tree snapshot.
- Manual keyboard testing exercises focus order, operability and visible focus in one pass. It needs no tooling and no assistive technology, and it finds a large share of what automation misses.
- Screen-reader testing exercises the thing that is actually delivered: what a user hears, in what order, and whether it changes when it should. Different screen readers and browsers pair differently, and the same markup can be announced differently by each (Semantics Before ARIA).
What this makes the browser do
And which of it is avoidable.
- Computing the accessibility tree is work the browser already does; a rule check reads that tree plus computed style for contrast, so per-state checking is cheap at the component level (Style Calculation).
- Running checks on a full page repeatedly during an end-to-end run is much more expensive and much less attributable than running them per component per state (Component Testing).
- Contrast rules need resolved colours, so they need real style resolution — one of the checks that a simulated document cannot do faithfully.
- Assistive technology consumes the tree across a process boundary. When the main thread is blocked the tree stops updating, and a screen reader reads stale content with no indication that it is stale (The Multi-Process Browser).
What a rule engine can and cannot decide
The line is not arbitrary. A rule fires when a property of the tree is unambiguously wrong: an element with no accessible name, an ARIA attribute that is invalid for its role, an insufficient contrast ratio, a duplicated id that breaks an association. These are decidable from a snapshot, and a good engine finds them reliably.
Everything on the other side of the line requires knowing what the interface is *for*. Is this name the one the user will say to their voice control? Does focus, after this action, land where the person needs it? Was that announcement comprehensible or did it interrupt? No amount of rule refinement makes these derivable, which is why the ceiling is where it is rather than where the roadmap is.
| Question | Decidable from the tree? | Who answers it | What a failure looks like to a user |
|---|---|---|---|
| Does this control have an accessible name? | Yes | Automated rule check, per component state | A screen reader announces "button" with no indication of what it does |
| Is the name meaningful and does it match the visible text? | No | Human review, and voice-control testing | The user says the label they can see and nothing happens |
| Is this ARIA attribute valid for this role? | Yes | Automated rule check | The control is announced as something it is not, or ignored entirely |
| Does the custom control respond to the keys its role implies? | No | Manual keyboard pass | A listbox that a keyboard user cannot move through |
| Is text contrast sufficient? | Yes, with resolved styles | Automated rule check in a real browser | Text that is unreadable in bright light or with low vision |
| Is focus visible at every step? | Partly — the rule sees a suppressed outline, not the whole journey | Manual keyboard pass, plus a visual snapshot of the focus state | The user tabs and cannot tell where they are (Visual Regression Testing) |
| Does focus go somewhere sensible after this action? | No | Manual keyboard pass | A dialog opens and the keyboard is still behind it |
| Is this status change announced, once, comprehensibly? | No | Screen-reader testing | The save succeeded and the user has no way to know |
| Can the whole task be completed without a pointer? | No | Manual keyboard pass over the full flow | The user reaches step three and stops |
The manual keyboard pass, as a checklist
This is the highest-value hour in accessibility testing and it requires no assistive technology, no tooling and no training: put the pointer away and complete the task. Run it on every flow that changed, and write down what you did so it can be repeated by someone else.
Do it on the real flow, including the failure paths. Most of the defects below only exist in states people do not think to visit — an expanded menu, a validation error, a dialog opened from inside another dialog.
- 1Tab from the top
Move forward through the whole page and confirm every interactive element is reachable, in an order that matches the visual order.
fails by A control that is skipped entirely — usually a
divwith a click handler — or an order that jumps around because of positioning (Div Soup: How It Happens and What It Costs). - 2Watch focus at every stop
Confirm there is always a clearly visible indicator, including on custom controls and on dark backgrounds.
fails by An outline removed by a reset, or an indicator with too little contrast against the element it is on.
- 3Activate with the keys the role implies
Enter and Space on buttons, Enter on links, arrows within composite widgets like tabs, menus and listboxes.
fails by A custom control that only responds to a click event, so it is reachable but not operable (Keyboard Events).
- 4Open and close every overlay
Confirm focus moves into the overlay, is contained while it is open, and returns to the trigger on close.
fails by Focus left on the page behind, or dropped to the document start on close so the user restarts the journey.
- 5Try to get stuck
Look for anything that traps focus with no keyboard exit — embeds, editors, third-party frames.
fails by A frame that swallows Tab, leaving the only escape as closing the tab.
- 6Trigger the error path
Submit invalid data and confirm the error is reachable, associated with its field, and that focus moves somewhere useful.
fails by An error shown only as colour, or announced nowhere, or placed above a scroll position the user never returns to (Errors People Can Actually Perceive).
- 7Change route and continue
Navigate and confirm focus moves to the new content rather than staying on a control that no longer exists.
fails by Focus stranded on the old page, so the next Tab starts from the top of the document (Client-Side Routing).
- 8Finish the task
Complete the entire journey — including submission and confirmation — without touching the pointer once.
fails by Any step that cannot be completed. This is the only pass/fail result that matters.
If a step cannot be completed, stop and fix it before continuing. Everything after a blocked step is untested regardless of what the automated report says.
Listening to what is actually announced
Screen-reader testing answers questions nothing else can: what is announced, in what order, with what wording, and whether a change reaches the user at all. It is slower than the keyboard pass and it needs the pairing written down, because the same markup is announced differently by different combinations.
The spec below is a small, concrete thing to test rather than a general aspiration — a form that saves. It contains the four failures that recur most: an unlabelled control, a status nobody hears, an error that is visual only, and focus that goes nowhere after the action.
semantics Native controls with visible labels associated by label; a submit button named by its visible text; a status region with an appropriate live role for progress and success; validation errors associated with aria-describedby and marked with aria-invalid.
| Tab | Reaches each field and the submit control in the order the form is read; nothing interactive is skipped. |
| Enter | Submits from within a text field, the platform behaviour people expect and a common gap in custom forms (Submission: Method, Encoding and Doing It Once). |
| Space | Activates the submit button when it is focused, and toggles checkboxes without submitting. |
| Escape | Dismisses any dialog the flow opens, returning focus to whatever opened it. |
- — On submit, focus stays somewhere stable rather than being lost when the button is disabled — disabling a focused element moves focus to the document unless it is handled.
- — On failure, focus moves to the first invalid field, or to an error summary that links to it.
- — On success, focus moves to whatever the person should do next, or stays put if nothing has changed structurally.
- — That saving has started — announced, not only shown as a spinner, which has no accessible presence unless you give it one.
- — That the save succeeded, once, without interrupting anything the person was reading.
- — On failure, what went wrong and what to do about it, associated with the field so it is repeated when that field is focused.
usually broken by The pattern invites a live region that is inserted at the same moment its text is set. The region did not exist in the tree when the content changed, so nothing is announced — and because it depends on update timing, it works in some frameworks and some states and not others, which is how it survives review (Live Regions and Announcement).
How to build it
Most important first.
- Run rule checks at the component level, in every state the component has — default, focused, invalid, loading, empty, expanded. Most violations concentrate in states that only exist mid-interaction (Loading, Error, Empty — The States You Did Not Render).
- Add rule checks at a few points inside critical end-to-end flows as well, because some violations only exist in composition: duplicate landmarks, two elements with the same name, a heading level skipped between components (Document Structure and Reading Order).
- Make the keyboard pass a written, repeatable checklist rather than an intention, and run it on every flow that changes. It is the highest-yield manual work available.
- Schedule screen-reader testing as recurring work with a named owner, on the pairing your users are most likely to have, and write down which pairing was used (Accessible Component Patterns).
- Test with real user settings: increased text size, reduced motion, high contrast, and a narrow viewport. Each is a supported configuration and each breaks different things (Contrast, Colour and Motion, Responsive Typography).
- Fix causes, not violations. When a checker reports a missing name, the fix is usually a real element with visible text, not an attribute that silences the rule (Semantics Are Behaviour).
- Include people who use assistive technology daily in real usability testing. Nothing above substitutes for that, and an engineer with a screen reader open for the first time is measuring their own unfamiliarity as much as the product.
Keyboard, focus, semantics, announcement
A required field on every lesson in this domain, not a section added when there is room.
- This lesson's whole subject is accessibility, so the field carries the part that is easiest to lose: the goal is task completion, not rule compliance. The question is always "can this person finish what they came to do", and no tool answers it.
- The keyboard pass has a fixed shape and finds a great deal: reach every control with Tab, see where focus is at all times, activate with Enter and Space, escape from every trap, and complete the whole task without touching the pointer (Keyboard Operability).
- Focus management across route changes and dialogs is manual by nature, because it is about sequence: where focus was, where it went, and whether the user can get back (Focus Management).
- Announcement testing means listening: does the status change reach the user, in a comprehensible form, without interrupting them and without repeating (Live Regions and Announcement).
- Forms are where these failures concentrate: an error must be announced, associated with its field, reachable, and phrased so it says what to do (Errors People Can Actually Perceive).
What can go wrong
- Treating a zero-violation report as a pass. It means no rule fired, and the rules were only ever the decidable third.
- Fixing to satisfy the rule: an
aria-labelthat does not match the visible text, breaking voice control for everyone who says what they can see. aria-hiddenapplied to make a violation disappear, removing content from the tree entirely rather than making it correct.- A checker run only on the initial state of a page, so the dialog, the error state and the expanded menu are never evaluated at all.
- The mitigation failing: a manual checklist that becomes a ritual, ticked without being performed, which produces the same false confidence with more effort.
- Screen-reader testing performed once, on one pairing, at the end of a project — a snapshot of one configuration presented as general coverage.
- Rules disabled wholesale because a legacy area produces noise, which quietly turns the check off for the new code too.
- A live region populated in the same update that inserts it may announce nothing, because the region did not exist in the tree when the text arrived. It is timing-dependent and reads as "sometimes it announces" (Live Regions and Announcement).
- Focus moved programmatically while the framework is still re-rendering can land on an element that is about to be replaced, sending focus to the document body a moment later (Reconciliation and Keys).
- When the main thread is busy, the accessibility tree lags the visual state, so a screen reader can announce the previous state while the screen shows the current one (Long Tasks).
- Accessibility testing has little direct attack surface, but the tempting shortcuts have consequences:
aria-hiddenon a container to silence a rule can hide a security-relevant warning from assistive technology while leaving it visible to everyone else. - Audit artefacts — recordings, screenshots, transcripts of screen-reader output on seeded pages — contain whatever data was on screen, and are frequently shared more widely than the data warrants (Session Replay and the Privacy It Costs).
- Where accessibility conformance is a contractual or regulatory obligation, the audit record is evidence. Overstating automated coverage in it is a reporting failure with real consequences.
- A third-party embed you cannot fix — a payment frame, a captcha, an ad — is part of your page's accessibility and part of its trust boundary at the same time (Third-Party Scripts and the Supply Chain).
- "We run the checker in CI, so we are accessible." You have covered the decidable fraction. The undecidable part is where people get stuck.
- "A third is close enough." The missing portion is not a random sample. It is concentrated in operability, focus and meaning — the parts that decide whether the task can be completed at all.
- "Automated tools will get there eventually." Some of what is missing is not a tooling limitation. Whether an accessible name is *meaningful* is a judgement about content, and better rules do not make it decidable.
- "Manual testing means checking it with a screen reader." The keyboard pass finds more, faster, with no tooling, and it is the one to do first.
- "An overlay widget fixes accessibility." It changes the page at runtime without fixing the semantics underneath, and it frequently breaks the assistive technology people already have configured (Semantics Before ARIA).
- "Our design system components are accessible, so our app is." Composition introduces its own defects: duplicated names, broken heading order, focus that moves between two individually correct components (Accessible Component Patterns).
Measuring it, and what changes in the field
- Track automated violations by rule and by component, so a rising count points at a pattern rather than at a page.
- Track manual pass completion per critical flow and the date of the last screen-reader session, including which browser and screen-reader pairing was used.
- Count defects found by the manual pass that the automated check did not report. That ratio is the local answer to "how much does automation cover here", and it is more useful than any published figure.
- Track accessibility issues reported by users, and check whether an automated rule could ever have caught each one. It usually could not, which is the argument for the manual budget.
- On a device without a physical keyboard, the operable path is different again: touch, switch access and voice control each have their own requirements, and none is exercised by a keyboard pass (Pointer Events).
- On a slow device, a blocked main thread stops the accessibility tree updating, so announcements arrive late or not at all — a failure with no visual counterpart (Long Tasks).
- With a screen reader on a different platform, the same markup can be announced differently. A finding on one pairing is evidence, not a general conclusion.
- At high zoom or large text, layout reflows and controls that were reachable can end up clipped or off screen (Responsive Typography).
- In a client-rendered application, nothing exists in the tree until the bundle has run, so the whole experience before hydration is part of what needs testing (Client-Side Rendering, Hydration).
- The manual work is genuine, recurring human time. There is no version of this where a tool absorbs it, and pretending otherwise is how it stops happening.
- Screen-reader testing has a learning curve, and early sessions measure the tester's unfamiliarity as much as the product. That cost is real and it is front-loaded.
- Running rule checks per component in every state slows the component suite. It is still much cheaper than finding the same violation in an end-to-end run or in production.
- Blocking a build on violations creates pressure to satisfy the rule quickly, which is exactly the pressure that produces attribute-shaped fixes. The gate needs review alongside it.
- Testing across several screen-reader and browser pairings multiplies effort and is the only way to know the product works on more than one.
Where this applies
Frontend advice ages badly and fragments across engines. These labels say what each claim is specific to, and where a different browser, device or framework would differ.
- GENERALThat automated rules can only decide what is derivable from the accessibility tree is structural, not a limitation of any one engine, and it holds for every browser and every rule set.
- SPEC-EVOLVINGThe tooling landscape moves: rule sets gain checks, guideline versions add and reword success criteria, and browser accessibility-tree implementations change what is exposed. Treat any specific coverage fraction, rule name or conformance mapping stated here as current-at-writing and check the present guidance and tool documentation rather than this text.
- PLATFORM-SPECIFICScreen readers are tied to platforms and pair differently with browsers — NVDA and JAWS on Windows, VoiceOver on macOS and iOS, TalkBack on Android, Orca on Linux — and the same markup can be announced with different wording, different verbosity or not at all, so a result on one pairing does not generalise to the others.
Where the depth lives
This domain teaches the browser-side mechanism and hands the rest off.
- — Testing & Reliability Engineering — the general form of this lesson is the limit of automated verification: what a checker can decide, what needs a human judgement, and how to budget for the part that cannot be automated instead of quietly dropping it.
- — Software Design — most accessibility defects are design decisions made before any code existed: a control that has no name because nothing in the design gave it one, or a flow that assumes a pointer.