Web Platform Test Running
Running 1.2 million conformance tests across 4 browsers every 24 hours
Web Platform Tests contains a large number of tests. When run in series on modern hardware, the entire suite requires over 24 hours to execute. We needed to factor the task into discrete segments, orchestrate parallel execution, and consolidate partial result sets prior to uploading to a central data store. All of this needed to operate reliably on top of systems that were in constant flux. From the layout engines, to the service providers, to the automation technology that powered the infrastructure (i.e. WebDriver), novel challenges could arrive at any moment.
We designed and deployed web-platform-tests/results-collection, a system built entirely from open source tooling including Ansible, Buildbot, and Terraform. This allowed us to promote test environment consistency across browsers, to display the status of every test publicly and in real time, and to gracefully recover from intermittent failures. All of our work was itself open source and well-documented, ready to be redeployed by anyone at any time.
The system has been reliably publishing results for all tested browser configurations since its deployment in March of 2018. This has allowed the Chrome team at Google to publish regular reports of browser interoperability to wpt.fyi.
More work like this at Bocoup
P.O. Box 961436
Boston, MA 02196