Pull Request Validation for Web Platform Tests

Google

Improving Test Author Workflow

View project

We worked with specification authors and browser implementers to develop a shared CI tool-chain for new test submissions, and a dashboard for viewing the the results in the four major browsers.

Challenge

As the number and diversity of contributors to the Web Platform Tests project grew, so too did expectations around its capabilities for verifying submissions. The project automatically verified the stability of each new test when run in theFirefox andChrome browsers. Developers from Microsoft and Apple were particularly interested in results for theEdge and Safaribrowsers. However, the existing test infrastructure was not well-suited to automate those browsers. Furthermore, the reporting mechanism for test results was already suffering from problems of scale; this enhancement would exacerbate the problem by increasing the amount of information that needed to be conveyed.

Solution

We first researched the available external services that could run these build jobs, ultimately selecting a free plan offered by Sauce Labs. Next, we extended the build infrastructure to support scheduling external validation, gathering the results, and reporting them alongside results from locally-run tests.

We then built a web application that made an overview of test results visible on GitHub.com and that also published a complete, well-formatted report. We deployed this service and made it a priority to formalize the process of "secret sharing," so that administrative responsibilities could be distributed between the existing Web Platform Tests maintenance team (and managed into the future).

Impact

We added support for Safari and Edge in May 2017, and launched the PR results application in August. As as a result, pull requests adding or modifying tests in the canonical Web Platform Tests repository are now run in CI in each browser. These results are reported back to the author and the reviewer in a clear and concise format. This has significantly improved the author/reviewer experience, sped up the review process, and increased velocity of the project.

I can't imagine ever achieving a reliably interoperable web platform without this.

Rick Byers, Google

Contact Us

We'd love to hear from you. Get in touch!

Boston

201 South Street, Boston, MA 02111

New York

315 Church St, New York, NY 10013

Phone & Email

(617)379-2752 hello@bocoup.com