Web Platform Test Running

Google

Running 1.2 million conformance tests across 4 browsers every 24 hours

Google's Web Platform Predictability project funded us to develop infrastructure for running the entire Web Platform Tests suite against the four major browser engines every day.

Challenge

Web Platform Tests contains a large number of tests. When run in series on modern hardware, the entire suite requires over 24 hours to execute. We needed to factor the task into discrete segments, orchestrate parallel execution, and consolidate partial result sets prior to uploading to a central data store. All of this needed to operate reliably on top of systems that were in constant flux. From the layout engines, to the service providers, to the automation technology that powered the infrastructure (i.e. WebDriver), novel challenges could arrive at any moment.

Solution

We designed and deployed web-platform-tests/results-collection, a system built entirely from open source tooling including Ansible, Buildbot, and Terraform. This allowed us to promote test environment consistency across browsers, to display the status of every test publicly and in real time, and to gracefully recover from intermittent failures. All of our work was itself open source and well-documented, ready to be redeployed by anyone at any time.

Impact

The system has been reliably publishing results for all tested browser configurations since its deployment in March of 2018. This has allowed the Chrome team at Google to publish regular reports of browser interoperability to wpt.fyi.

Contact Us

We'd love to hear from you. Get in touch!

Phone

+1 617-379-2752

Mail

P.O. Box 961436
Boston, MA 02196