Test Management App for ARIA-AT

In 2019, we worked with the W3C's ARIA-AT community group to design a screen reader interoperability test suite. The ARIA-AT project has an impressive goal: make assistive technologies and browsers render the accessible, semantic content of a webpage as reliably as they render its visible UI. In the fall, Facebook asked us to help design an assertion model and language for describing a test, but there was plenty more work to do.

Unlike other parts of the web platform, the interaction between assistive technologies (ATs) and browsers is not driven by a common API, making automated testing difficult. Nevertheless, the community wanted to collect interoperability data across time between different browsers and ATs running as many types of web components as possible, so we needed to come up with an approach that could scale to running tens if not hundreds of individual tests on a regular basis.

We also needed to coordinate the test authoring and test running process with a large number of stakeholders. In order to be successful, the project needed to collect feedback and incorporate multiple perspectives from W3C community group members, assistive technology vendors, browser implementers, and spec writers. We needed a way to facilitate different stakeholders providing feedback at different stages of the end-to-end testing process, from early test authoring to the review of draft results for publication. Consensus was particularly difficult for this project given that there is often no one answer for the right assistive technology behavior in a given situation.

We designed, built, user-tested, and deployed ARIA-AT App, a W3C-hosted test management application that defines and runs manual tests of assistive technologies to measure their support of accessibility semantics. The application allows project administrators to solicit and track new test runs from testers running specific browser and AT combinations. The application also provides an accessible interface for running the actual tests and resolving conflicts by browsing and raising issues on GitHub.

In the process of designing an app for the ARIA-AT community group, we also came up with a novel, accessible way of sharing in-progress UX designs. In addition to making traditional visual wireframes, we wrote textual “wireframes” which described the information hierarchy of each page along with the buttons and actions that they triggered.

In addition to the app itself, we helped create a Working Mode for the community group which defines various roles for stakeholders and provides guidance on how they can effectively provide feedback on all stages of the test authoring and test running process. We also coordinated a pilot test of the application that helped improve the UX of the ARIA-AT App and raise issues in the underlying tests for the community group to address.