Until recently, the execution of Test262 tests was consolidated into Webkit’s
./Tools/Scripts/run-jsc-stress-tests runner; which overtime had grown to be a “kitchen sink” for test running. This year Bocoup worked with WebKit to decouple the execution of Test262 tests from
run-jsc-stress-tests by creating a dedicated Test262 test runner (
test262-runner). This enables faster development (and build bot!) cycles during the feature implementation process.
The Test262 import process has also been re-designed (
test262-import). The resulting import routine was optimized to eliminate the manual burden of updating thousands of entries in a single index file after each import. This manual update was necessary to maintain the list of tests that should be skipped, instead of run, by the original Test262 runner.
Before we dig into the actual tools, let’s do a quick orientation in a local copy of the WebKit source. Since the WebKit source repository is quite large, with an extensive history (202,744 commits as of this writing), we recommend creating a shallow clone:
git clone email@example.com:WebKit/webkit.git webkit --depth=1 && cd webkit
git clone git://git.webkit.org/WebKit.git webkit --depth=1 && cd webkit
Once the cloning is complete, familiarize yourself with the following directory locations in the project:
- Test262 is imported to the
./JSTests/test262folder; helper files can be found in
./JSTests/test262/harnessand the actual test files are in
- The Test262 import and runner scripts (
test262-runner) are located in the
./Tools/Scriptsfolder, with dependencies located in
Ensuring that Test262 import is consistent and reliable is only part of the solution. Import operations must also reduce the human effort burden, by automating as much administrative work as possible.
Operationally, the import script will fetch the master branch of Test262 published to the official repository at
https://github.com/tc39/test262. The changes are applied in the
./JSTests/test262 folder, along with additional information:
test262-Revision.txtis updated to store the latest import revision (the commit hash of Test262) and the source from the last import.
latest-changes-summary.txtwill store a summary of the latest imported files, including status codes: (A) added, (M) modified, (R) renamed and (D) deleted files. This information is also useful to the runner if the user wants to check only the newly imported files.
(Although it’s not recommended, Test262 can be imported from a local folder, using the
--src argument, ie.
./Tools/Scripts/test262-import --src <folder>. The script can also import from a custom remote git source, ie.
./Tools/Scripts/test262-import --remote <url>.)
To update WebKit’s local copy of Test262, execute
./Tools/Scripts/test262-import. The approximate expected output of that operation will look similar to the following:
Settings: Remote: firstname.lastname@example.org:tc39/test262.git Branch: master -------------------------------------------------------- Importing Test262 from git > git clone -b master --depth=1 email@example.com:tc39/test262.git /var/folders/xz/k2lxgs3s43180_tjwn1v_kwc0000gn/T/khgeCKEdmd Cloning into '/var/folders/xz/k2lxgs3s43180_tjwn1v_kwc0000gn/T/khgeCKEdmd'.. remote: Counting objects: 37388, done. remote: Compressing objects: 100% (15828/15828), done. remote: Total 37388 (delta 22766), reused 29039 (delta 21422), pack-reused 0 Receiving objects: 100% (37388/37388), 13.00 MiB | 1.85 MiB/s, done. Resolving deltas: 100% (22766/22766), done. Checking out files: 100% (36321/36321), done. New tracking: firstname.lastname@example.org:tc39/test262.git From branch: master New revision: 0fde488bb4cddccdc154b4cd913cb19d940102f6 Summary of changes: M harness/atomicsHelper.js M harness/detachArrayBuffer.js M harness/features.yml M harness/nans.js M harness/nativeFunctionMatcher.js M harness/proxyTrapsHelper.js M harness/regExpUtils.js M harness/tcoHelper.js M harness/testAtomics.js M harness/testIntl.js A test/annexB/language/comments/single-line-html-close-unicode-separators.js M test/built-ins/Atomics/Symbol.toStringTag.js M test/built-ins/Atomics/add/bad-range.js A test/built-ins/Atomics/add/bigint/bad-range.js A test/built-ins/Atomics/add/bigint/good-views.js A test/built-ins/Atomics/add/bigint/nonshared-int-views.js M test/built-ins/Atomics/add/descriptor.js A test/built-ins/Atomics/add/expected-return-value.js M test/built-ins/Atomics/add/good-views.js M test/built-ins/Atomics/add/length.js M test/built-ins/Atomics/add/name.js M test/built-ins/Atomics/add/non-views.js M test/built-ins/Atomics/add/nonshared-int-views.js M test/built-ins/Atomics/add/shared-nonint-views.js [Snip!] > rm -rf /path/to/webkit/JSTests/test262/harness > rm -rf /path/to/webkit/JSTests/test262/test > mv /var/folders/xz/k2lxgs3s43180_tjwn1v_kwc0000gn/T/khgeCKEdmd/harness /path/to/webkit/JSTests/test262 > mv /var/folders/xz/k2lxgs3s43180_tjwn1v_kwc0000gn/T/khgeCKEdmd/test /path/to/webkit/JSTests/test262 Done in 37 seconds!
After updating Test262, a new commit to the WebKit source repository is necessary. Operators will want to run
./Tools/Scripts/test262-runner to check for any new outcomes.
When called with no arguments, this script will run every test imported from Test262, with the exception of those files included in the skip list. The skip list is defined in
./Tools/Scripts/test262-runner −−skipped−files will run all of the skipped tests and flag any newly passing tests.
Additional options and flags for
test262-runner can be found by executing
Upon execution, the runner will read a list of test files that are expected to fail from
./JSTests/test262/expectations.yaml, and report their latest outcome. If any new failure is found, the runner will report them as new failures and will close the program with a non-zero exit code.
To run a subset of tests, e.g. limited to
./Tools/Scripts/test262-runner -o test/built-ins/ArrayBuffer. The approximate expected output of that operation will look similar to the following:
Settings: Test262 Dir: JSTests/test262 JSC: WebKitBuild/Debug/jsc Child Processes: 32 DYLD_FRAMEWORK_PATH: /path/to/webkit/WebKitBuild/Debug Paths: test/built-ins/ArrayBuffer Config file: JSTests/test262/config.yaml Expectations file: JSTests/test262/expectations.yaml --- 156 tests run 2 test files skipped 22 tests failed in total 0 tests newly fail 0 tests newly pass Saved all the results in /path/to/webkit/test262-results/results.yaml Summarizing results... See the summaries and results in the /path/to/webkit/test262-results. Done in 4.80 seconds!
./Tools/Scripts/test262-runner --save and then committing the changes.
The expectations file is a machine generated file that doesn’t allow tracking the reason or bugs referencing the failure, e.g. an un-implemented Stage 3 feature. It’s recommended to triage new failures and add them to the skip list with a matching WebKit Bugzilla link using a comment line. Note that the expectations file exists primarily to unblock updates of Test262 into WebKit.
Some common usages include:
- Running tests from a specific file or folder, execute
./Tools/Scripts/test262-runner -o <path>. This option can be stacked to multiple paths:
./Tools/Scripts/test262-runner -o <path1> -o <path2>.
- Triaging new failures from a recent Test262 import, use the option to run only the recently added and modified test files:
- Initiating a complete test run, including the tests that would normally be skipped, execute:
By default, the
./Tools/Scripts/test262-runner −−jsc <path-for-jsc>; this path will also be used to try and set the environment’s
- The expected folder similar to calling
- The expected folder similar to calling
- A path found calling
By default, the
test262-runner uses 4 child processes per core to execute a test run. If the target machine has 4 cores available, it will use 16 children processes. If only 1 core is available, it will use 4 processes. To set a custom number of cores, the runner should be invoked as
./Tools/Scripts/test262-runner -p <number>, with the desired number of cores to be used.
test262-runner is done running the tests, it creates a “git ignored” folder which contains the summaries and reports that were output by the latest run. This folder is named
test262-results and is placed in the current folder where the runner was called.
test262-results folder may contain the following files:
index.html: an HTML report with a short summary and list of failures. It includes all the failures, not only the new failures.
summary.html: presenting two tables of summaries of the results per path – folders and subfolders – and features from the frontmatter metadata.
report.css: used in both HTML files.
results.yaml: a long Yaml file with all the results, can be consumed by any script.
summary.txt: a text version of the summaries.
summary.yaml: a Yaml file with the data used for the summaries