As planned, we began our work by focusing on the V8 engine’s existing ES2015 tests. Using a process built on excellent open source tools such asRecast andGitslave, we ported tests authored forV8’s “mjsunit” test suiteto run in the Test262 harness.
After that effort, it was on to writing brand new tests! This certainly
challenged our newfound specification-reading abilities. We stepped through
algorithm after algorithm writing tests to ensure that every function,
expression, and abstract operation behaves exactly as intended. This resulted
in substantial coverage for most of the new language features for ES2015, from
String.raw, generator expressions to destructuring
assignment, we covered a lot of ground.
To date, we’ve filed hundreds of bugs against Google’s V8 engine, Mozilla’s SpiderMonkey engine, various open source projects related to programming language processing, and even the ECMAScript specification itself! We’ve followed through with patches in many cases, submitting 36 spec patches and 14 V8 patches. The interplay between these efforts has been entertaining: sometimes, we remove specification text rather than write tests for it. At other times, our changes to the spec have made a lot more work for us when writing tests. For the curious, we maintain a wiki page to track our additional contributions.
This work occasionally exposes novel problems in the testing strategy. In these cases, we worked to extend the Test262 infrastructure to support the new tests. This included things such as denoting which tests need to be interpreted as ES2015 modules and ensuring that some tests run without modification. Most notably, we recognized a major hurdle in adequately testing features that are expressed by multiple independent syntactic forms. To address this, we conducted a months-long opendesign anddevelopment process, ultimately resulting in a process to procedurally generate certain sets of test files.
But it’s not just the big challenges that excite us. Day-to-day, we commonly write tests for the most bizarre code—code that, had we seen it during a “normal” code review, we would immediately request refactoring. Although we’ve found ourselves taking longer showers, writing this kind of code helps us identify bugs in the specification itself.
Best of all, since we started helping out, TC-39 enhanced the standardization process to require that proposals for new features include Test262 tests. So although it’s taken a huge investment of time and effort to bring Test262 “up to snuff” for ES2015, the commitee is, well, committed to preserving that coverage over time.
It still falls on all of us to honor that commitment, of course, so we want to help more people get involved! To that end, we’d love to revitalize the project’s dormant website and automate the publication of test results for various web browsers. We’re still full of ideas on how Test262 can be made more useful to more developers. We’re even thinking about how some tests might be procedurally generated from the specification text itself!
So really, this is just the beginning of a much larger effort. We don’t know where we’ll end up, but we’re excited to learn that alongside you.