Test Management App for ARIA-AT

Facebook

Building an app to manage manual testing of browsers and assistive technologies rendering ARIA accessibility features.

View project

We designed and built an app that manages manual test runs of web browsers and assistive technologies to help scale WPT-inspired interoperability tests for ARIA accessibility features.

Challenge

In 2019, we worked with the W3C's ARIA-AT community group to design a screen reader interoperability test suite. The ARIA-AT project has an impressive goal: make assistive technologies and browsers render the accessible, semantic content of a webpage as reliably as they render its visible UI. In the fall, Facebook asked us to help design an assertion model and language for describing a test, but there was plenty more work to do.

Unlike other parts of the web platform, the interaction between assistive technologies (ATs) and browsers is not driven by a common API, making automated testing difficult. Nevertheless, the community wanted to collect interoperability data across time between different browsers and ATs running as many types of web components as possible, so we needed to come up with an approach that could scale to running tens if not hundreds of individual tests on a regular basis.

We also needed to coordinate the test authoring and test running process with a large number of stakeholders. In order to be successful, the project needed to collect feedback and incorporate multiple perspectives from W3C community group members, assistive technology vendors, browser implementers, and spec writers. We needed a way to facilitate different stakeholders providing feedback at different stages of the end-to-end testing process, from early test authoring to the review of draft results for publication. Consensus was particularly difficult for this project given that there is often no one answer for the right assistive technology behavior in a given situation.

Solution

We designed, built, user-tested, and deployed ARIA-AT App, a W3C-hosted test management application that defines and runs manual tests of assistive technologies to measure their support of accessibility semantics. The application allows project administrators to solicit and track new test runs from testers running specific browser and AT combinations. The application also provides an accessible interface for running the actual tests and resolving conflicts by browsing and raising issues on GitHub.

In the process of designing an app for the ARIA-AT community group, we also came up with a novel, accessible way of sharing in-progress UX designs. In addition to making traditional visual wireframes, we wrote textual “wireframes” which described the information hierarchy of each page along with the buttons and actions that they triggered.

In addition to the app itself, we helped create a Working Mode for the community group which defines various roles for stakeholders and provides guidance on how they can effectively provide feedback on all stages of the test authoring and test running process. We also coordinated a pilot test of the application that helped improve the UX of the ARIA-AT App and raise issues in the underlying tests for the community group to address.

Impact

The ARIA-AT App has allowed the W3C ARIA-AT CG to accelerate their ability to author and run tests. For the first time, the community group can gather interoperability data that they can use to get buy-in from vendors and browser implementers about the highest-priority accessibility fixes.

We see ARIA-AT App as a key piece of infrastructure in the accessibility ecosystem. Just as WPT allows browser implementers and web developers to quantify interoperability for web platform features and test262 allows ECMAScript engine implementers and developers to do the same for JavaScript, ARIA-AT will eventually quantify the interoperability of browsers and assistive technologies for ARIA accessibility features. Building an easy way to collect test run data is the first step.

Working with Facebook and the W3C ARIA-AT CG, we believe the ultimate potential of ARIA-AT goes much further. Our vision for ARIA-AT, in tandem with the ARIA Authoring Practices Guidelines, is to enable full interoperability of platform-level accessibility features and to build an ecosystem of accessibility tools and guidance, including high-level web components that would make it easy for any web developer to build upon a platform that is inclusive and accessible by default.

Two layered screenshots of the Aria-AT app, each in its own illustrated browser window. The first is of a the home page. The second is of draft results for checkbox example tested with VoiceOver for macOS Safari

Contact Us

We'd love to hear from you. Get in touch!

Mail

P.O. Box 961436
Boston, MA 02196