One of the best parts of the Scratch community is the diversity of Scratch projects. Community members have used the Scratch programming language to create many different kinds of interactive applications, from full game engines to music sequencers. One genre is especially unique: Multiple Animator Projects, or MAPs. These Scratch projects compile animations from many different users into one collaborative file, often set to music. One of the most popular MAPs, the Seagulls MAP, contains over 300 images, composed of multiple “sprites” with many “costumes” or frames! As you can imagine, loading 300 images on the web is no small task, especially if they need to be used in a WebGL rendering engine like the Scratch renderer.
While we’ve previously posted about Scratch 3 optimizations to achieve smooth frame rates, today we’d like to focus on a different aspect of performance: load times. We’ve been experimenting with a new method to substantially improve load times for Scratch projects, especially those, like MAPs, which include hundreds of sprites and costumes.
First, we’ll outline the existing process for loading images in a Scratch project and then we’ll explain our new experiment which uses a “texture atlas” to pack multiple assets into a single bitmap. We’re especially excited by this new implementation because it harkens back to the days of early computer graphics systems, when many sprite images were combined into a single texture in order to improve memory locality and reduce draw calls. Hop in, we’re going back to the future!
Loading images in the Scratch VM
Every time you create a sprite in Scratch 3 or draw a new “costume” frame in the project editor, Scratch creates a new image bitmap and adds it to your project. Scratch also supports uploading SVG sprites, which may contain embedded image files. As mentioned above, a single Scratch project may incorporate hundreds of bitmaps among other assets, including sound files. If you export your Scratch 3 project as a .sb3 file, you can even unzip the project to see that Scratch stores each bitmap as a separate image file (or use our in-progress sb-util API to query all of the image assets in a project).
We’ve previously described how the web-based Scratch player uses WebGL to handle scene rendering and sprite compositing. Loading a project therefore requires unzipping the .sb3 file clientside with jszip and creating WebGL textures for every sprite/costume. This process requires a few steps for each image. First, each sprite is loaded with an HTMLImageElement object (or an ImageBitmap, if supported) in order to draw the image to a 2d Canvas element. Finally this canvas element can be used to create a WebGL texture. Initializing new Image elements for each asset can quickly become a bottleneck in loading a project with hundreds of sprites. Not only does this process take a significant amount of time, but it also incurs memory overhead for each JavaScript object / DOM element.
An idea from the past, repurposed for the present
In order to make the process of loading images more efficient, we decided to experiment with creating a “texture atlas” for all of the images in a project. You may be familiar with texture atlases by way of video game atlases or CSS sprite sheets. The basic concept of combining many small images into a larger image has been used for a variety of different optimization purposes since the earliest days of computer graphics. Video game texture atlases were first created to overcome limitations of existing rendering systems, often allowing different images to be rendered to the screen using a shader to sample different parts of a single memory-efficient texture. CSS sprite sheets, meanwhile, bundled small images together for a very different reason—to avoid the bandwidth and memory penalties of fetching many small files over the network. Our approach uses a texture atlas to avoid the runtime startup cost of instantiating unnecessary JavaScript objects, DOM elements, and their callbacks.
This new approach requires two modifications to the Scratch VM: serializing a project’s images to the texture atlas on file save and deserializing the texture atlas on load. The serialization process involves drawing all of the image assets as tiles on a single, large canvas and then saving the canvas as a texture atlas image alongside JSON metadata that describes the coordinates of each original image tile in the atlas. The original image files can be kept in the project for backwards compatibility, at the expense of a slight increase in filesize.
Below you can see a snippet of how we manage tiles as part of the texture atlas:
class MapTile {
constructor ({width, height, top, left}) {
this.width = width;
this.height = height;
this.top = top;
this.left = left;
}
getImageData (mapAsset) {
return mapAsset.getImageData(this.left, this.top, this.width, this.height);
}
putImageData (mapAsset, data) {
mapAsset.putImageData(this.left, this.top, this.width, this.height, data);
}
}
class MapArea {
constructor ({subarea = null, width, height, left = 0, top = 0}) {
this.subarea = subarea;
this.left = left;
this.top = top;
this.width = width;
this.height = height;
}
findFreeArea (width, height) {
if (this.subarea) {
const areaInSub = this.subarea.findFreeArea(width, height);
if (areaInSub) return areaInSub;
}
if (this.width >= width && this.height >= height) {
const {left, top} = this;
this.subarea = new MapArea({subarea: this.subarea, width, height: this.height - height, left, top: top + height});
this.left += width;
this.width -= width;
return new MapTile({width, height, top, left});
}
return null;
}
}
The deserialization process involves loading the texture atlas into a single HTMLImageElement and creating an ImageBitmap to draw the entire atlas to a single canvas. Then, the VM can use the JSON metadata along with CanvasRenderingContext2D.getImageData() to read individual tiles from the canvas and create WebGL textures for each original asset. In the case of a project with hundreds of assets, this completely eliminates the need to instantiate hundreds of HTMLImageElements, ImageBitmaps, and Canvas elements.
We also implemented an additional experiment to improve the speed of loading SVG assets. Since many Scratch SVG assets contain embedded bitmap images, we also parse SVG files on save and add any embedded bitmap images to the project’s texture atlas. Then we export the SVG (sans embedded bitmap) in a JSON representation using paper.js and save it alongside the atlas. When the Scratch project is later loaded, we restore the embedded bitmaps from the atlas into the serialized paper.js tree and render directly to a Canvas element.
Further testing and optimization
Our texture atlas experiment is still in its early days. Right now we’ve just implemented the optimization on a branch of the Scratch VM. We are planning to stress test the idea by measuring load times across a number of popular Scratch projects. We also need to balance the improvement in load times against other factors, including the effect of large project files on network download speed and unzipping time.
Down the road, we may explore further improvements, including sampling the texture atlas directly from the GPU, although this approach would prove challenging as we would have to support texture bleeding and work around other quirks of the way that shaders sample textures.
In the meantime, we will continue looking for ways to support and optimize creative use cases like Multiple Animator Projects. Projects that push the boundaries of the Scratch 3 engine are exactly what make the community so special!
