Scratch 3 Motion Detection and Sound Control

MIT

Writing a web based augmented reality and sound control system for Scratch 3.

Scratch is a visual programming language and online community which empowers children to create their own stories, games, and animations. The Scratch team brought us on to implement the video sensing and re-implement scratch-audio for the the forthcoming release of Scratch 3, a ground up rewrite of the Scratch virtual machine and Editor in Node.js and React.

Challenge

The Scratch team needed to re-implement Scratch 2 audio and video functionality for the scratch 3 release. This required knowledge of web video and web audio programming interfaces, web performance and scratch internals.

Solution

We ported the Scratch 2 ActionScript implementation of video sensing to JavaScript for Scratch 3, making JavaScript performance enhancements along the way. We then worked on the `scratch-audio subsystem using the Web Audio API and common interactive audio tricks the interactive audio programming experience "sound smooth" in a cross browser way.

Impact

As a result of this work, Scratch projects that depend on Audio or Video Sensing will be able to get the scratch functionality they need in order to run in Scratch 3.

We were also able to use the Scratch blocks now supported in Scratch 3 to make a pretty neat A/R synth demo:

You can run the above example in your own browser by downloading the rhodes-video.sb3 Scratch file, and loading it into the Scratch 3 editor here: llk.github.io/scratch-gui/develop (look in the "Scratch bar" at the top of the browser window for "File" => "Upload from your computer").