Automated Accessibility Testing: Exposing the Myth & Supercharging Your Product

You’ve been lied to about automated accessibility testing. Its impact has been exaggerated in hopes of hooking you with the shiny benefit of securing impeccable accessibility easily. But in QA services, there are no shortcuts. There’s skill, determination, and hard work.

So, here’s what you can really do with automated accessibility testing tools and how you can use them meaningfully.

Can Accessibility Testing Be Automated?

Yes. But it won’t be the magical solution you’re hoping for. Automated accessibility testing is very different from your standard automation of, say, functional tests. The core difference is the limited powers of AT tools in this context. No software can comprehensively evaluate how comfortably a differently-abled person would navigate your service.

You’ve probably seen a few articles with sections titled “What is automated accessibility testing”. And the explanation following it is akin to: “It’s the process of ensuring that your product is usable by any person, regardless of their physical or mental abilities.” But that’s an exaggeration.

Most automated accessibility testing tools can only find basic issues:

• Missing alt text.
• Empty links.
• Color contrast concerns, etc.

These are certainly helpful, as they facilitate the work for your QA experts. Yet, automated tools only won’t ensure that any person can enjoy your app to the fullest. Because fixing such aspects is only scratching the surface. For a genuinely all-user-friendly project, you’ll need manual software testing from experts who’re truly skilled in digital accessibility.

But don’t be discouraged. Automated testing services can still help you quite a lot here. They can speed up the SDLC, unload your team, and handle some time-consuming things. You just shouldn’t hope for a tool that’ll take care of everything perfectly. So, if we’re talking about manual vs automated accessibility testing, the former wins. As there’s no substitute for human judgment and perspective.

Accessibility Aspects That Suit Automation Best

For accessibility, automated testing is, without a doubt, beneficial. But only when you use it in moderation and in a productive manner. For that, you’ll need to consider two things:

1. Whether certain elements should be automated at all.
2. And how well the automated tools will handle a particular accessibility feature.

So, first, you ought to evaluate if AT will be beneficial. And then apply automation where it’s needed via an approach that’s actually valuable. For example, you can decide to fully automate something. Or combine manual and automated testing for optimal results.

Here are some accessibility areas where test automation services are most useful.

• Verifying the correct usage and hierarchy of structural elements such as headings, lists, and landmarks.
• Identifying missing or inadequate alternative text.
• Detecting color combinations that don’t meet accessibility standards.
• Validating the presence of labels associated with form input fields.
• Evaluating whether all interactive elements, such as links and buttons, can be accessed and operated with a keyboard.
• Confirming that each page has a unique and descriptive title with proper language attributes.
• Securing proper focus management.
• Checking for the appropriate use of semantic HTML elements.
• Assessing the accessibility of interactive widgets, such as sliders and tabs.
• Validating the correct usage of ARIA (Accessible Rich Internet Applications) roles and attributes.

As you can see, AT can cover quite a few things. But for more intricate scenarios, like checking semantic meanings, usability, and support of advanced assistive technologies, you’ve got to go with manual testing.

Challenges of Automating Accessibility Testing

In most cases, companies combine both manual and automated accessibility testing services. And that’s a good thing. Since AT can only reliably locate up to 30% of issues. So, it’s incredible to see businesses not compromising quality and doing what needs to be done for happy users.

Let’s discuss core reasons why automated accessibility testing tools may not always provide the outcomes you’d expect. It’ll also help you visualize where manual QA expertise should be applied.

Dynamic & Interactive Content

Applications often contain dynamic and interactive content (e.g., dropdown menus that alter with time or customer preferences). For AT tools, it’s difficult to accurately assess the accessibility of such elements, as they don’t have the contextual awareness for the modification.

Lack of Context Awareness

Automated tools can’t fully grasp the context in which content is presented. For example, they may flag an image without alt text, even if it’s purely decorative and does not convey meaningful information. Thus, AT software may create a problem out of thin air.

False Positives & Negatives

With the above factors, false positives are also rather common for automated accessibility testing tools. And, unfortunately, they may lead to wasted time and effort. Developers will have to investigate and address erroneous results. Plus, your trust in the effectiveness of AT software might take a hit.

Incompatibility with Assistive Technologies

To offer pleasant experiences to users with disabilities is to understand them. And automated tools lack this ability. So, they may not accurately simulate the behavior of people who rely on screen readers or voice recognition software. Thus, automated tests may not reflect real-world accessibility concerns.

Limited Scope of Testing

In general, automated tools can only adequately assess specific accessibility aspects. But they’re incapable of covering all relevant criteria. As a result, even if you decide to automate accessibility tests, you can do so only with a handful of them.

You should also remember that automated web accessibility testing is more mature than, say, automated accessibility testing for mobile apps. The latter have unique traits, such as gesture-based navigation or screen rotation, that complicate the testing process.

And this further highlights the need for manual QA.

Maintenance Overhead

Just like any script, automated accessibility tests require ongoing maintenance. You ought to update and modify them to keep their relevance. While this element is not unique, you should understand that it’s a bit tougher to do for accessibility. Plus, guidelines and standards like WCAG may shift as well.

Accessibility Expertise Requirement

It’s surprising that some resources say that automated accessibility testing tools can be used by people with little to no skill. As a QA company with over a decade of experience, we have to say that nothing in QA services should be approached like this. As a kid can rely on the best art supplies in the world and still make a mess.

It’s not about the equipment. It’s about how you operate it. So, don’t view AT software as a silver bullet. It’s a good mallet for quickly hammering down accessibility issues. But whether you hit the right nails or the wrong fingers is up to the person who’s wielding it.

If you want to get remarkable results for your accessibility testing, whether automated or manual, strive to cooperate with experienced accessibility testing services. They won’t disappoint you. Won’t waste your time and money. And will surprise you with the quality of your product after their work is done.

Benefits of Automated Accessibility Testing

AT software is made for procedures that are very common. It allows us to save time, reduce costs, and free up the teams. So, the fact that automated accessibility testing tools exist tells us one important thing: digital accessibility is becoming a staple. And that is just phenomenal.

It’s phenomenal that companies are moving away from “products for profit” and to “products for people”. Because it’s users who decide the fate of your project (and their satisfaction also secures higher revenue). In the future, we hope to see automated accessibility testing software evolve and mature.

Yet, even at the current development level, it has a few perks to offer, as it:

• Allows for quick scanning of content for accessibility issues.
• Provides consistent results across multiple runs.
• Helps identify accessibility issues early in the development process.
• Can be scheduled to run regularly, enabling continuous monitoring of accessibility compliance.
• Is well-suited for large-scale projects with numerous web pages or complex applications.
• Can cover a decent range of accessibility criteria.
• Promptly generates detailed reports documenting accessibility issues and suggested remediation steps.
• Can be integrated into development workflows and CI/CD pipelines.
• Reduces the need for manual testing efforts, saving time and resources.
• Empowers developers to identify and address accessibility issues early, fostering a culture of accessibility awareness and compliance.

So, though you are most likely to have to mix both automated and manual accessibility testing, as is often the case, it’s just better to have both. They provide distinct benefits. Each with their own positive impact on user happiness.

When to Automate Accessibility Testing

“When to automate accessibility testing” is a tricky question. As it can’t be fully automated. At least not to the degree it’s usually done with other types of tests. So, here, instead of pinpointing a particular time slot, look for indicators that signal you could benefit from automation.

Hence, you should consider automated accessibility testing when:

• Testing turns repetitive and needs to be performed frequently. It’ll let your team focus on value-adding tasks and speed up delivery.
• Your project involves a large number of web pages or complex applications. You’ll be able to cut down on time-consuming manual checks and improve testing consistency.
• The product is early in the development cycle. Automated tests allow you to catch defects promptly and avoid costly retrofitting.
• You need to run routine compliance checks. AT tools can offer ongoing assessments to ensure continued compliance.
• Consistency in accessibility evaluations is essential. For multi-team or multi-environment projects, automated accessibility testing tools can supply uniform results across different setups.
• You need to ensure that accessibility improvements made in previous iterations don’t introduce new issues. Automated tests can be run as part of regression testing efforts.

You could also automate accessibility testing as a part of your scaling efforts. In other words, you might not need AT right now. But it’s much better to get everything prepared in advance than panic later on.

Automated Accessibility Testing Tips

Once you’re confident in your decision to automate accessibility testing, you’ll have to decide on your approach to it. We won’t tell you how exactly you should do it. It’s a highly personalized decision and there’s no doubt you know your project best. Thus, as always, our QA experts share insights from their practice on things you should consider for refined results.

#1 Understand Accessibility Guidelines & Educate Your Team

Yes, AT tools will locate certain accessibility issues for you. But if you don’t understand what they mean, why did they happen, and how to properly fix them, well, you get the idea. To avoid such instances, don’t rely solely on automated accessibility testing tools.

Make sure your team is knowledgeable in accessibility standards. They will be able to accurately interpret and prioritize results. Amend any errors in a meaningful manner. And strengthen their skills and your product.

#2 Choose the Right Tools

The “right” tool isn’t necessarily the one with only five-star reviews and a big user pool. These, of course, wouldn’t hurt. But when selecting automated accessibility testing tools, you need to focus on:

• The technologies they support and whether they’ll cover what you’re working with (web, mobile, etc.).
• The features they offer and whether they’re able to handle your needs.
• The learning curve, which is especially significant for teams with in-development expertise.
• And the budget you’re currently operating with.

Another point you ought to direct your attention to is the tool’s community. As you’ll be able to turn to other consumers for assistance and gain quick, first-hand insights.

#3 Run Regular Scans

Turn accessibility test runs into a standard assessment. Schedule automated scans regularly to catch new issues introduced by code changes. It’ll let you prevent regressions and secure ongoing compliance. You can also incorporate automated accessibility testing into your development process.

#4 Customize Testing, Analyze & Prioritize Results

Configure your automated testing tool to match your specific accessibility requirements. Adjust parameters such as:

• Standard being checked against.
• Severity levels.
• And rulesets to align with your goals.

Prioritize fixing issues based on severity and impact on users. Use tools that provide detailed reports and prioritization features to streamline this process.

#5 Verify Results Manually

Don’t forget about the value of human input. Conduct manual accessibility testing alongside automated checks to ensure thorough coverage. You don’t have to do it for everything. If you’re confident AT tools can cover certain aspects with no issues – leave them be. Your team will thank you for it.

But if you feel that particular accessibility scenarios would benefit from manual QA – don’t hesitate to engage extra expertise.

#6 Include Diverse Users

Remember that automated tools can’t fully replicate the experience of differently-abled users. So, from our practice, we’d advise to include people with diverse abilities in your testing process. You can work with QA engineers who have disabilities. Conduct research or crowd-testing for authentic feedback. And implement users’ insights for refined quality.

#7 Continuously Improve

Accessibility is an ongoing process. Hence, you shouldn’t stop at what you have. Don’t be satisfied with just being satisfied. Strive to review and refine your automated testing strategy based on:

• Team and consumer feedback.
• Developments in accessibility technology.
• And changes to accessibility standards.

One tiny improvement in your product can make a big difference.

Accessibility Testing Process

Automated accessibility testing isn’t much different in terms of processes. So, you won’t have any extra surprises or worries here. The only distinction you’ll face is the need for ample knowledge of the accessibility norms you’re aiming for.

Preparation Stage

The first phase should be fully dedicated to product familiarization. And your team needs to become its therapist. That is knowing the project’s special features, goals, weaknesses, and fears.

Project Analysis

Ensure your entire team, from stakeholders to developers, is on the same page about product specifics. Everyone needs to understand your project’s unique features, weak points, and relevant accessibility guidelines. Without this shared awareness, the testing process may get messy.

Selection of AT Strategy & Tools

Outline approaches, techniques, tools, and resources that will be engaged in automated accessibility testing. Together, they will serve as a coherent roadmap towards desired results.

Creation & Approval of Test Documentation

Keep detailed records of project documentation. This includes what you expect from the testing process, how everything will be done, etc. Here, it’s beyond critical to seek stakeholders’ involvement. It’ll ensure there are no mismatches between business expectations and technical execution.

Communication Channel Validation

Supply your team, whether in-house or external, with protocols regarding communication. Define what platforms will be used. How and when the crew is to connect. And what’s the procedure for urgent issue resolution.

Active Stage

This phase directly links with the previous one. What you have prepared before will decide how the active stage will perform. So, if you’re having any troubles here, come back and reevaluate what’s already present.

1. Create a resilient testing framework that matches the selected automation approach and tools.
2. Write and ensure your scripts are in accordance with authorized documentation.
3. Specify precise parameters and configurations for running test suites or individual tests.
4. Integrate mechanisms to produce comprehensive reports detailing the results of test runs.
5. Execute tests prioritizing user-valued, high-risk, and critical areas.
6. Ensure that your team thoroughly documents any defects or anomalies uncovered during testing.

Support Stage

And the last phase is all about enjoying your wins and learning from your mistakes. You’ll also plan and gear up for your future endeavors.

1. Create exhaustive instructions for executing tests and troubleshooting.
2. Regularly review test reports and validate test results for accuracy and reliability.
3. Continuously update and maintain automated scripts to adapt to changes.
4. Expand coverage to include newly developed features.

Automated Accessibility Testing Tools

And now, let’s review a few tools for automated accessibility testing. Note that we’re not saying you should use these ones, they’re the best, etc. No. There’re a few options that have proven quite useful, including in our practice. And you could rely on these as a guide as to what you should look for.

Deque Axe (aXe)

• Offers browser extensions for manual testing and developer tools for integration into CI/CD pipelines.
• Provides guidance and suggestions for fixing accessibility problems.
• Can be integrated with development tools for code linting and reviews.
• Offers configuration options and integrates with various testing frameworks and CI/CD tools.

WAVE

• Offers a free, web-based interface. Simply enter the URL of your website and it’ll generate a report highlighting potential accessibility errors and warnings.
• Categorizes issues by severity (errors, contrast warnings, etc.) and provides clear descriptions of each problem.
• Is a lightweight tool that performs scans quickly. This is ideal for preliminary checks or during development cycles where frequent testing is needed.
• Incorporates a built-in contrast checker, which helps identify insufficient color contrast between text and background elements.

Google Lighthouse

• Comes pre-integrated into Chrome DevTools.
• Offers accessibility audits alongside other performance checks like SEO and page speed.
• Categorizes accessibility issues by severity (critical, serious, etc.), helping developers prioritize which problems need immediate attention.
• While it primarily functions as an integrated tool within Chrome DevTools, it’s also open-source.

Pac11y

• Is an open-source library, allowing developers to integrate it into their existing testing frameworks and customize it for specific needs.
• Can be used for testing in various environments, including browsers, Node.js, and command-line tools.
• Provides detailed reports on identified accessibility issues, including code snippets and explanations.
• Integrates with popular build tools like Grunt and Gulp.

Siteimprove Accessibility

• Can scan thousands of pages simultaneously, providing a comprehensive overview of your entire website’s accessibility posture.
• Prioritizes accessibility issues based on severity and WCAG conformance levels (A, AA, AAA).
• Can visually highlight accessibility issues directly on the webpage.
• Tracks your website’s accessibility compliance over time.

To Sum Up

To make your product accessible is to prioritize the users. You ought to focus on solving their pains, not ease project development. And that means doing what’s right instead of what’s simple. So, don’t hesitate to rely on automated accessibility testing. But supplement it with human judgment when needed and engage experts who care about your product as much as they care about your consumers.