Ensuring Correctness of W3C Accessibility Examples

Facebook

Writing a regression test suite for the WAI-ARIA Authoring Practices Examples.

Facebook funded Bocoup to write comprehensive regression tests for the Web Accessibility Initiative Accessible Rich Internet Applications (WAI-ARIA) Authoring Practice Guideline Examples.

Challenge

The Authoring Practices Guidelines (APG) help web developers gain a deeper intuition for the purpose and use of the WAI-ARIA specification. This is done via the definition of design patterns that use features in web browsers specified by the WAI-ARIA specification. These design patterns show how modern day, interactive web applications can be accessible to screen readers with the use of ARIA roles, attributes and expected key handlers for keyboard interaction.

The APG includes HTML and JavaScript examples for the design patterns to help web developers get started. The example implementations are frequently complex, with a multitude of keyboard interactions and necessary attributes. The APG team feared that future contributions to these example implementations could introduce hard-to-notice bugs. A call was made for assistance in writing comprehensive regression framework and easily expandable tests of the APG examples. Bocoup, experts in testing open web standards, answered this call.

Solution

We worked closely with the W3C ARIA Authoring Practices Task Force to produce a testing framework that consumes the examples on each example page in the documentation. We wrote tests in Node.js using the open source AVA framework and Selenium WebdriverJS to hook into the browser, allowing for interactive tests of the examples. Over the course of three months, we wrote 583 tests for 40 example widgets, and produced extensive documentation of the framework, test norms, and use of AVA and SeleniumJS.

All tests run on TravisCI to provide continuous integration testing for future work on the Authoring Practice Guide. In addition to the tests, we created an automated report, which runs on every pull request and details how comprehensive the regression tests for a given example page are.

Impact

The tests written by Bocoup have already caught and documented 33 bugs in the current draft of the Authoring Practice Guide examples. The practice guideline examples for the WAI-ARIA specification promote accessibility in keeping with our core values of a open and accessible web platform. Catching regressions in these examples will help keep best practices up to date, and prevent errors in the canonical examples from proliferating onto the web. The regression test suite Bocoup provided will also help the ARIA Authoring Practices Task Force confidently accept work from new contributors going forward.

Contact Us

We'd love to hear from you. Get in touch!

Phone

+1 617-379-2752

Mail

P.O. Box 961436
Boston, MA 02196