Advancing the Open Web through Test262

In the beginning of 2015, we started another important project for the open web. Of course you haven’t forgotten, but for all the folks just joining us: Google and Bocoup teamed up to improve Test262, the official test suite for the JavaScript language. Our goal was to improve the dependability of the web platform (not just the V8 engine) as a base for application development. We focused on improving coverage for new features being introduced by the then-unfinished ES2015 specification, but we cast a wide net for ways to add value. The eighteen months that followed have been crammed full of adventure, discovery, and beauty. We filed hundreds of bugs on V8 and SpiderMonkey, and landed dozens of patches to V8 and to the specification itself. There’s even been some derring-do! In this post, we’ll recap some of that magic and share our ideas on how to prolong it.

As planned, we began our work by focusing on the V8 engine’s existing ES2015 tests. Using a process built on excellent open source tools such as Recast and Gitslave, we ported tests authored for V8’s “mjsunit” test suite to run in the Test262 harness.

After that effort, it was on to writing brand new tests! This certainly challenged our newfound specification-reading abilities. We stepped through algorithm after algorithm writing tests to ensure that every function, expression, and abstract operation behaves exactly as intended. This resulted in substantial coverage for most of the new language features for ES2015, from @@iterator to String.raw, generator expressions to destructuring assignment, we covered a lot of ground.

To date, we’ve filed hundreds of bugs against Google’s V8 engine, Mozilla’s SpiderMonkey engine, various open source projects related to programming language processing, and even the ECMAScript specification itself! We’ve followed through with patches in many cases, submitting 36 spec patches and 14 V8 patches. The interplay between these efforts has been entertaining: sometimes, we remove specification text rather than write tests for it. At other times, our changes to the spec have made a lot more work for us when writing tests. For the curious, we maintain a wiki page to track our additional contributions.

This work occasionally exposes novel problems in the testing strategy. In these cases, we worked to extend the Test262 infrastructure to support the new tests. This included things such as denoting which tests need to be interpreted as ES2015 modules and ensuring that some tests run without modification. Most notably, we recognized a major hurdle in adequately testing features that are expressed by multiple independent syntactic forms. To address this, we conducted a months-long open design and development process, ultimately resulting in a process to procedurally generate certain sets of test files.

But it’s not just the big challenges that excite us. Day-to-day, we commonly write tests for the most bizarre code—code that, had we seen it during a “normal” code review, we would immediately request refactoring. Although we’ve found ourselves taking longer showers, writing this kind of code helps us identify bugs in the specification itself.

Our work has been highly collaborative. We coordinate with members of the V8 team on a daily basis. We attend the meetings regularly held by TC-39, the standards body responsible for the language (and Leo recently joined Rick as a fully-fledged committee member). We routinely work with JavaScript language enthusiasts on the project issue tracker and the es-discuss mailing list. The experience has been exciting, challenging, and enriching. Working with Google to improve test coverage for V8 has allowed us to contribute to the stability of the open web platform substantially, far beyond what we were able to do while we were focused on other projects.

Best of all, since we started helping out, TC-39 enhanced the standardization process to require that proposals for new features include Test262 tests. So although it’s taken a huge investment of time and effort to bring Test262 “up to snuff” for ES2015, the commitee is, well, committed to preserving that coverage over time.

It still falls on all of us to honor that commitment, of course, so we want to help more people get involved! To that end, we’d love to revitalize the project’s dormant website and automate the publication of test results for various web browsers. We’re still full of ideas on how Test262 can be made more useful to more developers. We’re even thinking about how some tests might be procedurally generated from the specification text itself!

So really, this is just the beginning of a much larger effort. We don’t know where we’ll end up, but we’re excited to learn that alongside you.