Bocoup recently facilitated an update to the WebKit project’s interaction with Test262. In this article, I’ll cover what this means for the WebKit project and the JavaScript ecosystem, as well as what exactly has been done in the WebKit project to help make this process more repeatable.

Test262 is a project maintained by Ecma’s TC39. Test262 is home to conformance tests for JavaScript’s language syntax and built-in APIs, as specified by ECMA-262, ECMA-402 and ECMA-404. The test material and harness code is entirely written in JavaScript.

“WebKit is the web browser engine used by Safari, Mail, App Store, and many other apps on macOS, iOS, and Linux.”1 WebKit includes an ECMAScript implementation called JavaScriptCore. When implementing a new feature in JavaScriptCore, Test262 is used to verify conformance to the specification of that feature.

Until recently, the execution of Test262 tests was consolidated into Webkit’s ./Tools/Scripts/run-jsc-stress-tests runner; which overtime had grown to be a “kitchen sink” for test running. This year Bocoup worked with WebKit to decouple the execution of Test262 tests from run-jsc-stress-tests by creating a dedicated Test262 test runner (test262-runner). This enables faster development (and build bot!) cycles during the feature implementation process.

The Test262 import process has also been re-designed (test262-import). The resulting import routine was optimized to eliminate the manual burden of updating thousands of entries in a single index file after each import. This manual update was necessary to maintain the list of tests that should be skipped, instead of run, by the original Test262 runner.

Before we dig into the actual tools, let’s do a quick orientation in a local copy of the WebKit source. Since the WebKit source repository is quite large, with an extensive history (202,744 commits as of this writing), we recommend creating a shallow clone:

git clone git@github.com:WebKit/webkit.git webkit --depth=1 && cd webkit

Or

git clone git://git.webkit.org/WebKit.git webkit --depth=1 && cd webkit

Once the cloning is complete, familiarize yourself with the following directory locations in the project:

  • Test262 is imported to the ./JSTests/test262 folder; helper files can be found in ./JSTests/test262/harness and the actual test files are in ./JSTests/test262/test.
  • The Test262 import and runner scripts (test262-import and test262-runner) are located in the ./Tools/Scripts folder, with dependencies located in ./Tools/Scripts/test262.

test262-import

Ensuring that Test262 import is consistent and reliable is only part of the solution. Import operations must also reduce the human effort burden, by automating as much administrative work as possible.

Operationally, the import script will fetch the master branch of Test262 published to the official repository at https://github.com/tc39/test262. The changes are applied in the ./JSTests/test262 folder, along with additional information:

  • test262-Revision.txt is updated to store the latest import revision (the commit hash of Test262) and the source from the last import.
  • latest-changes-summary.txt will store a summary of the latest imported files, including status codes: (A) added, (M) modified, (R) renamed and (D) deleted files. This information is also useful to the runner if the user wants to check only the newly imported files.

(Although it’s not recommended, Test262 can be imported from a local folder, using the --src argument, ie. ./Tools/Scripts/test262-import --src <folder>. The script can also import from a custom remote git source, ie. ./Tools/Scripts/test262-import --remote <url>.)

To update WebKit’s local copy of Test262, execute ./Tools/Scripts/test262-import. The approximate expected output of that operation will look similar to the following:

Settings:
Remote: git@github.com:tc39/test262.git
Branch: master
--------------------------------------------------------

Importing Test262 from git
> git clone -b master --depth=1 git@github.com:tc39/test262.git 
/var/folders/xz/k2lxgs3s43180_tjwn1v_kwc0000gn/T/khgeCKEdmd

Cloning into '/var/folders/xz/k2lxgs3s43180_tjwn1v_kwc0000gn/T/khgeCKEdmd'..
remote: Counting objects: 37388, done.
remote: Compressing objects: 100% (15828/15828), done.
remote: Total 37388 (delta 22766), reused 29039 (delta 21422), pack-reused 0
Receiving objects: 100% (37388/37388), 13.00 MiB | 1.85 MiB/s, done.
Resolving deltas: 100% (22766/22766), done.
Checking out files: 100% (36321/36321), done.
New tracking: git@github.com:tc39/test262.git
From branch: master
New revision: 0fde488bb4cddccdc154b4cd913cb19d940102f6

Summary of changes:
M harness/atomicsHelper.js
M harness/detachArrayBuffer.js
M harness/features.yml
M harness/nans.js
M harness/nativeFunctionMatcher.js
M harness/proxyTrapsHelper.js
M harness/regExpUtils.js
M harness/tcoHelper.js
M harness/testAtomics.js
M harness/testIntl.js
A test/annexB/language/comments/single-line-html-close-unicode-separators.js
M test/built-ins/Atomics/Symbol.toStringTag.js
M test/built-ins/Atomics/add/bad-range.js
A test/built-ins/Atomics/add/bigint/bad-range.js
A test/built-ins/Atomics/add/bigint/good-views.js
A test/built-ins/Atomics/add/bigint/nonshared-int-views.js
M test/built-ins/Atomics/add/descriptor.js
A test/built-ins/Atomics/add/expected-return-value.js
M test/built-ins/Atomics/add/good-views.js
M test/built-ins/Atomics/add/length.js
M test/built-ins/Atomics/add/name.js
M test/built-ins/Atomics/add/non-views.js
M test/built-ins/Atomics/add/nonshared-int-views.js
M test/built-ins/Atomics/add/shared-nonint-views.js
[Snip!]

> rm -rf /path/to/webkit/JSTests/test262/harness

> rm -rf /path/to/webkit/JSTests/test262/test

> mv /var/folders/xz/k2lxgs3s43180_tjwn1v_kwc0000gn/T/khgeCKEdmd/harness 
/path/to/webkit/JSTests/test262

> mv /var/folders/xz/k2lxgs3s43180_tjwn1v_kwc0000gn/T/khgeCKEdmd/test 
/path/to/webkit/JSTests/test262

Done in 37 seconds!

After updating Test262, a new commit to the WebKit source repository is necessary. Operators will want to run ./Tools/Scripts/test262-runner to check for any new outcomes.

test262-runner

When called with no arguments, this script will run every test imported from Test262, with the exception of those files included in the skip list. The skip list is defined in ./JSTests/test262/config.yaml. Changes in this file are not automated. A human must add or remove tests to be skipped. The skip list is also documentation of tests which are failures, mostly due to known—and linked—bugs or new features not yet implemented in JavaScriptCore. The skip list can list test files by their path or using features tags, which correspond to metadata defined in each of the Test262 test files. Executing ./Tools/Scripts/test262-runner −−skipped−files will run all of the skipped tests and flag any newly passing tests.

Additional options and flags for test262-runner can be found by executing ./Tools/Scripts/test262-runner --help.

Upon execution, the runner will read a list of test files that are expected to fail from ./JSTests/test262/expectations.yaml, and report their latest outcome. If any new failure is found, the runner will report them as new failures and will close the program with a non-zero exit code.

To run a subset of tests, e.g. limited to ArrayBuffer, execute ./Tools/Scripts/test262-runner -o test/built-ins/ArrayBuffer. The approximate expected output of that operation will look similar to the following:

Settings:
Test262 Dir: JSTests/test262
JSC: WebKitBuild/Debug/jsc
Child Processes: 32
DYLD_FRAMEWORK_PATH: /path/to/webkit/WebKitBuild/Debug
Paths: test/built-ins/ArrayBuffer
Config file: JSTests/test262/config.yaml
Expectations file: JSTests/test262/expectations.yaml
---


156 tests run
2 test files skipped
22 tests failed in total
0 tests newly fail
0 tests newly pass
Saved all the results in /path/to/webkit/test262-results/results.yaml
Summarizing results...
See the summaries and results in the /path/to/webkit/test262-results.

Done in 4.80 seconds!

With new changes from the JavaScriptCore source or with new updates from Test262, it’s important to record these files in the skip list, or as new failures in the expectations file. This can be done by executing ./Tools/Scripts/test262-runner --save and then committing the changes.

The expectations file is a machine generated file that doesn’t allow tracking the reason or bugs referencing the failure, e.g. an un-implemented Stage 3 feature. It’s recommended to triage new failures and add them to the skip list with a matching WebKit Bugzilla link using a comment line. Note that the expectations file exists primarily to unblock updates of Test262 into WebKit.

Some common usages include:

  • Running tests from a specific file or folder, execute ./Tools/Scripts/test262-runner -o <path>. This option can be stacked to multiple paths: ./Tools/Scripts/test262-runner -o <path1> -o <path2>.
  • Triaging new failures from a recent Test262 import, use the option to run only the recently added and modified test files: ./Tools/Scripts/test262-runner −−latest−import.
  • Initiating a complete test run, including the tests that would normally be skipped, execute: ./Tools/Scripts/test262-runner --ignore-config.

Environment

By default, the test262-runner will try to detect the path for JavaScriptCore, but it’s also possible to provide a custom path calling it with ./Tools/Scripts/test262-runner −−jsc <path-for-jsc>; this path will also be used to try and set the environment’s DYLD_FRAMEWORK_PATH (if not yet defined). The default JavaScriptCore path is detected in the following order, returning an error if it doesn’t find JavaScriptCore:

  • The expected folder similar to calling webkit-build-directory --debug
  • The expected folder similar to calling webkit-build-directory (release)
  • A path found calling which jsc

By default, the test262-runner uses 4 child processes per core to execute a test run. If the target machine has 4 cores available, it will use 16 children processes. If only 1 core is available, it will use 4 processes. To set a custom number of cores, the runner should be invoked as ./Tools/Scripts/test262-runner -p <number>, with the desired number of cores to be used.

When test262-runner is done running the tests, it creates a “git ignored” folder which contains the summaries and reports that were output by the latest run. This folder is named test262-results and is placed in the current folder where the runner was called.

The test262-results folder may contain the following files:

  • index.html: an HTML report with a short summary and list of failures. It includes all the failures, not only the new failures.
  • summary.html: presenting two tables of summaries of the results per path – folders and subfolders – and features from the frontmatter metadata.
  • report.css: used in both HTML files.
  • results.yaml: a long Yaml file with all the results, can be consumed by any script.
  • summary.txt: a text version of the summaries.
  • summary.yaml: a Yaml file with the data used for the summaries

Conclusion

With these new changes in place, JavaScriptCore implementers now have a more reliable set of tools for importing and running Test262 in their target environments. We’re excited to streamline this process as part of an effort to improve runtime interoperability by getting JavaScript implementers on-board with sharing test material more often. In future posts, we’ll step through the process we’ve been working on for exporting WebKit test material back into shared test suites.

  1. webkit.org ↩