WPT: An overview and history

A few of our colleagues and partners have asked for an overview of WPT. We wrote this internal overview and history, which we want to also share publicly here as a reference for anyone interested in the how the project operates and came to be.

What is WPT?

Introduction

Web Platform Tests, stylized `web-platform-tests` or `WPT`, is an open-source cross-browser-engine project founded in 2010 to test browser conformance to web standards. The project is actively maintained by standards developers from Apple, Bocoup, Google, Igalia, Microsoft, Mozilla, and the W3C through a governance structure described below. WPT has over 56,552 tests and counting (with 1.8 million subtests) at the time of this writing. Each test is run every couple of hours in Continuous Integration (CI) through the infrastructure behind wpt.fyi.

Impact

In 2010, browser interoperability was a mess. Web developers had to write multiple versions of the same code in order to get things to work in every browser, which often led to bugs, and bad experiences for people using the web. During the course of the 2010s, WPT was the center of a push toward making browsers interoperable. In order to make browsers render web code interoperably, all major browsers engines committed to run and pay attention to a new comprehensive automated conformance test suite called WPT.

WPT has been the operational venue through which browser companies collaborate on making their browsers more interoperable. Browser companies continue to collaborate through WPT to close the growing long tail of interoperability on the web.

Operating model

Core Team

A group of Core Team members preside over the project. This group of people was appointed in 2019. Appointment is by invitation. Interested parties can email core team members if they would like to join.

Decision Making

WPT builds consensus between browser engineering teams on changes to the test suite and governance model through a Request for Comment (RFC) process. Disagreements that cannot be solved through this process are escalated to the Core Team.

Interop group

The majority of the coordination in WPT is managed through an annual planning process called Interop. A group of about a dozen members from companies that run or contribute to browser engines meet bi-weekly throughout the year to coordinate and manage this process.

The group works in two ways:

1. First on interop issues, which are web platform features the group has agreed to focus on stabilizing, picked from community submitted proposals.
2. Second on investigation areas, which are web platform features that aren’t quite ready for interop because they require further investigation. The current investigation areas are Accessibility Testing, Mobile Testing, and WebAssembly Testing.

The Interop process is governed by an Interop Charter and follows roughly this sequencing:

1. In the fall, the Interop group launches a call for proposals from community members about what issues to focus on in the following year.
2. The group spends the fall narrowing down what, if any, issue and investigation areas will continue.
3. The group then creates a spreadsheet to map out all the community proposals.
4. Each company represented in the interop group fills out the spreadsheet with votes and vetoes.
5. The members meet to align on what issues to select for the following year.
6. The spreadsheet is finalized and at the beginning of the following year the interop group announces the selected issues and investigation areas through blog posts on each participating member company’s blog.

Two-Way Sync

Each participating browser company maintains a two-way sync with WPT through which new tests added to the browser engines are upstreamed to WPT, and in turn new tests added to WPT are downstreamed back into each browser engine. This results in a mass sharing of test material across engines. The tests are used in day to day work by browser engineers to evaluate conformance of their implementations of standards to the standards specifications.

Corporate form

WPT functions as an informal and independent project with no single company backing the project, and no contract in place between the participating companies. While the project is led, operated, and is in every sense close to standards organizations like the World Wide Web Consortium (W3C) and Web Hypertext Application Technology Working Group (WHATWG), there is no governing Corporation, as is the case for the W3C, or Steering Group Agreement, as is the case for the WHATWG.

WPT History

Pre 2010

WPT started off with the W3C HTML Working Group’s test suite for HTML called `html-testsuite`. This project was maintained by the HTML Testing Task Force members: Sam Sneddon, James Graham, Robin Berjon and Kris Krueger.

Much of the basic infrastructure for WPT, like testharness.js and an early version of reftests, existed as far back as 2010. Some of the test suites that were imported to the first version of WPT had content dating back to the late 90s.

During this time Sam Sneddon, James Graham, and Simon Pieters worked at Opera Software on the Opera browser test suite called T (released publicly as Presto Testo, named after Opera’s browser engine, “Presto”). James Graham created testharness.js to avoid repeated boilerplate in Opera’s JS Tests. Apple’s browser engine WebKit also had a common framework in their LayoutTests, and Mozilla’s Gecko engine had shared functions in Mochitests.

2010 - 2013

Robin was lead editor of the W3C HTML specification at the W3C at this time. His mandate was to make an interoperable, stable snapshot of HTML ready for Recommendation status, the final status of a W3C specification, by September 2014. This was hard to imagine doing at the time because test suites were fragmented and had low coverage¹. Fragmentation in particular made it hard to prove interoperability, and impossible to test the interactions between HTML and other specs. As a result, Robin began to focus on an integrated test suite in order to solve these problems.

Opera had a separate but related problem trying to automate fragmented and manual standards test suites so they could run in CI. At the time, Opera’s testing infrastructure involved screenshot reference testing, and was also very flaky.

Opera also wanted strong interoperability testing because they were a smaller browser with less market share, which caused fewer web developers to test their work in the Opera browser. During this period, Opera would occasionally change browser behavior to match Internet Explorer (IE), and at the same time IE would change behavior to match other browsers. Making a shared test suite would create a space to collaborate more on standards conformance. A shared test suite also promoted greater interoperability with larger browsers that web developers were testing in, meaning websites would work better in Opera.

In addition to the problems experienced by the W3C and Opera, browser vendors and W3C participants were lost as to how to contribute to or run tests, which was a major problem for actually implementing standards. The majority of standards conformance tests were written by browser developers, so getting their test material into the shared test suite made sense for a couple of reasons. For the W3C, a shared test suite of tests written by browser engineers provided a way to get to Recommendation status with HTML 5. For Opera, a shared test suite created a way to share tests across browser engine implementation teams and drive material interoperability.

There was also a rift between W3C HTML and WHATWG HTML, which had forked HTML based on a disagreement about the future of the platform, as well as differing working styles. There was a sense that migrating to a shared test suite and increasing interoperability velocity would facilitate better collaboration between W3C and WHATWG.

At the same time there was a growing desire among some members at the W3C to shift power away from browser companies toward web developers by opening up testing through an initiative called Test The Web Forward, an initiative started by Adobe engineers who also wanted a single way of writing tests for contributing code to multiple engines. A shared and coherent test suite would open this door.

For all these reasons, a decision was made to integrate testing systems, but there was some work to do. Test suites were managed separately by each working group and some working groups no longer existed. A lot of the test suites had been written to “prove” that a specification had two implementations before shipping a Recommendation and not touched since. Some were in Mercurial. Some were in git on GitHub but not under the W3C org. Some were in a W3C hosted CVS. Some didn’t have licenses. Many used different ad hoc harnesses, some of which no longer worked.

Robin and Mike Smith from the W3C began importing test suites from across the W3C, trying to preserve history, and running them through Perl scripts to transform them into using testharness.js.

There was tension during this time as well. For example, Sam Sneddon was trying to improve one of the test suites at the W3C: the CSS Working Group CSS 2.1 tests. This was controversial because of concerns that attempting automation might delay getting CSS 2.1 to Recommendation status in the W3C Process, and that merging with other test suites might make it harder to audit and understand the test suite. These concerns were ultimately addressed in later years.

2013 - 2015

In 2013 other test suites started to merge into html-testsuite, and the group of contributors working on the test suite renamed the project from html-testsuite to web-platform-tests. Soon after that, the WebApps Working Group test suite was merged in. The decision to do this was made by rough consensus between the people most involved in the project. Folks were aligned on high-level goals, and the project fell into place.

In 2013, Tobie Langel became the W3C Testing Lead as a W3C Fellow on behalf of Facebook. Tobie wrote about WPT on the W3C blog. Tobie created the W3C vision for the open web platform testing effort, including an action plan, cost-estimate methodology, and related tooling. Tobie also played a key role in bringing the Test the Web Forward brand (based on Divya Manian and Paul Irish’s work on Move the Web Forward) to the W3C and facilitating the process of converging different working group contribution practices, review processes, and IP needs. During this period we began to see improvements in the ability to automate tests, run them more often with a new runner, and check their reliability across many runs.

2015 - 2017

In 2015, Sam started to work on the CSS Working Group test suite again. This took two years because the CSS Working Group both had the largest and most manual test suite. CSS tests merged into web-platform-tests in early 2017.

James Graham also introduced a stability checker in 2016. And Rick Byers from the Google Chrome team started encouraging test-driven approaches to spec writing.

2017 - 2019

Up until now, WPT had been a W3C repo. Philippe Le Hégaret, project manager at the W3C, wanted WPT to become its own independent project. His vision was that in order for web standards to be stable, they need three legs: 1) a specification, 2) two or more implementations, and 3) tests. Until this point testing had been decentered, and now that it was picking up steam, Philippe thought it should be independent from the W3C. So a new github org was created called web-platform-tests and WPT was moved to it. There was no governance set up, just an IP-neutral contributors license agreement. The people involved were the people in the room.

In 2017, Philippe le Hegaret wrote about the project on the W3C blog. The annual W3C conference that year, TPAC, also had a session called “Testing Project (aka Web Platform Tests)” in which representatives from Apple, Bocoup, Google, Microsoft, and Mozilla discussed how to advance the project.

There was a desire to have shared CI and a dashboard to report test results back to the browser companies. Participants were apprehensive that it could turn into a situation like the Acid test suite where browsers were pitted against each other to pass shallow feature tests. There was concern that this would lead to distorted incentives around contributing tests because the press would pick up these results, and organizations would only contribute tests they pass to make them look better than the competition. This would incentivise large numbers of surface-level tests, rather than anything from which vendors could derive interoperability signals.

At the TPAC Testing Project session Boaz Sender from Bocoup suggested that instead of reporting on the pass/fail rate of each browser, an interoperability metric could be computed. This way everyone wins as interoperability increases and there is no way for tech press to report negatively on the “worst scoring” browser. Rick Byers from Google liked this idea and funded Bocoup to operationalize the regular execution of tests and publication of results (these results would later drive wpt.fyi).

Mike Pennisi from Bocoup spent the next 2 years leading a team to build the infrastructure to run WPTs in automation every time a new browser version released, and working with Philip Jägenstedt who led the effort to integrate this work into the Google Chrome org. Philip collaborated with Brendan McLoughlin at Bocoup to implement two-way sync functionality between Chromium and WPT, so that browser engineers could run WPT as part of their workflow, and easily upstream the tests they were writing locally. Philip and Bocoup shared the source code and architecture for this sync process with engineers at Apple and Mozilla, who implemented the same.

This period also saw a big campaign at Google, lead by Rick Byers and Philip Jägenstedt, to train Google employees working in Chromium on how to write tests in the WPT format.

This was also the time during which Microsoft decided to switch to the Chromium browser engine. Microsoft also funded Bocoup to port their manual tests to WPT in order to be able to run them against Edge.

2019 - Present

In January 2019, active contributors to WPT met and established a Core Team including Sam Sneddon, James Graham (Mozilla), John Jansen (Microsoft), Mike Smith (W3C), Ms2ger (Igalia), Philip Jägenstedt (Google), Simon Pieters (Bocoup), and Youenn Fablet (Apple). Later that month, Philip Jägenstedt opened the first RFC for the WPT project to establish an RFC process.

In 2022, the Core Team started an annual interop process documented at the beginning of this writing. This process now runs annually to finish the growing long tail of web browser interoperability.