Exploring New Technologies for Making Maps (Part Two): Two Fragment Shaders and a Mouse

In part one of this series we started learning how to make maps rendered by WebGL, a browser based hardware-accelerated graphics API for 2D and 3D graphics. Our access to this technology was via Tangram, a map rendering library from Mapzen. This post will focus primarily on shaders, those perplexing parallel programs that power our pixels, and how to create visual effects using them. We will be focusing on 2D effects for now and will also look a bit at how to create interactive effects that are powered by shaders. If you haven’t read part one, you might want to start there.

What are shaders anyway?

Shaders are the not-so-secret sauce to creating hardware-accelerated performant real-time visual effects. They are programs, written in a c-like language called GLSL, that are used to compute the final color of a pixel on the screen. They make your graphics card’s rendering pipeline programmable.

There are two kinds of shaders, vertex shaders, which act on points of geometry; and fragment shaders, which operate on pixels (or fragments in WebGL speak). What makes them fast, and also difficult to write, is that the shader program is run for every pixel (or vertex) being rendered in parallel. However running in parallel means that each pixel doesn’t really know what’s going on with the other pixels.

We are used to being able to write loops to look at regions of the canvas and treat them as groups of related pixels; but in a shader program there is a sense that the whole world is just that pixel currently being processed. They are the Lilliput of graphics APIs: tiny, self contained and a bit oblivious to what is going on in the world around them.

The other thing that makes them fast is they run on specialized hardware, the GPU or graphics processing unit on your computer. However, this means that they are separated from the CPU and memory that we usually program with. Thus once running they are a bit harder to interact with than we might be used to.

Two resources for more details on this both come from a resource I found quite helpful in writing this post. “What is a fragment shader?” and “An Introduction for those coming from JavaScript” are both part of the Book of Shaders and are both great and friendly introductions to what a shader does and what GLSL is respectively. While this post will give a flavor of what it looks like to program with shaders, the Book of Shaders is something you’d want to dive into to get a better feel for the syntax of GLSL.

As mentioned, there are 2 kinds of shaders:

• Vertex Shaders: These are given the geometry being rendered (lets say a polygon or a cube) and can change the coordinates of points (vertices) that make up the geometry. This can be used to transform, scale, or otherwise mutate a shape.
• Fragment Shaders: These are given the fragments (pixels) that are going to be rendered to the screen for each piece of geometry. The fragment shader’s job is to assign a color to this pixel. You can do this in multiple passes to create effects such as light, shadow and material properties. This is the shader type we will focus on in this post.

Shaders in Tangram

So how can we take advantage of shaders in Tangram? A lot of Tangram’s built in rendering is built on shaders and the library allows us to tap into its rendering pipeline at various points. The diagram below, from the shaders overview for Tangram, demonstrates where we can jump in.

via Tangram Documentation

Here is what one of those fragment shaders looks like, remember that the Tangram configuration is written in YAML. We are going to create a custom style with the definition for our shader

styles:
  defines:
    shaders:
      TANGRAM_WORLD_POSITION_WRAP: false
  #coordinates_fragcoord is the name of the style
  coordinates_fragcoord:         
    base: polygons
    shaders:
      blocks:
        # the color block sets the color of the pixel. The contents of this block
        # is a fragment shader program and is written in GLSL
        color: |
          // Get the coordinate of this pixel and normalize it to the 0-1 range.
          vec2 coord = gl_FragCoord.xy / u_resolution;

          // The color variable represents the color that this pixel takes.          
          // Let us assign the x coordinate to the red channel and the 
          // y coordinate to the green channel. Set blue to 0.
          color.r = coord.x;
          color.g = coord.y;
          color.b = 0.0;

We’ll apply this style to the water layer and draw the earth layer in a solid color like this.

layers:
  earth:
    data: { source: osm }
    draw:
      polygons:
        order: 1
        color: '#222'

  water:
    data: { source: osm }
    draw:
      coordinates_fragcoord: # Use our custom style
        order: function() { return feature.sort_key; }

This will result in the following

Understanding the coordinate space of the shader

What can we tell from this image? If we look in the lower left corner we see it is pretty dark. This is where both the red and green channel gets assigned a zero value. The bottom right is where the x-coordinate is greatest and the y-coordinate is still zero, resulting in a pure red. The top left is pure green suggesting the y-coordinate is 1 and the x-coordinate is zero; and finally yellow is where red and green combine.

At some point in my coding I drew the following ascii diagram in a comment to remind me of what this coordinate space looks like (after normalization)

// (0,1)----(1,1)
//   |--------|
//   |--------|
//   |--------|
// (0,0)----(1,0)

Unlike the coordinate system in the 2D canvas API, the Y axis starts at the bottom and grows upwards (like regular cartesian coordinates).

Debugging in shaders

This brings me to another tip I picked up when writing these. Since these shader programs run on the GPU, we can’t just write things to the console to debug things (e.g. if you wanted to write out some property of the pixel currently being processed). Instead you want to find a way to express your test or debugging condition in terms of color. Here is a pattern I used a lot to understand the coordinate space, or debug things that weren’t showing up in the right place

color: |
  vec2 coord = gl_FragCoord.xy / u_resolution;

  color.r = coord.x;
  color.g = coord.y;
  color.b = 0.0;

  // Run a test and set the color to a sentinal color you can easily identify
  // In this case we were not using pure blue before.
  if(coord.x > 0.5) {
    color.r = 0.0;
    color.g = 0.0;
    color.b = 1.0;
  }

Setting any pixel with a coord.x value greater than 0.5 to blue results in the following

We can use the same technique to look at the range of glFragCoord

if(gl_FragCoord.x > 700.) {
  // Use white as a sentinal color
  color.r = 1.0;
  color.g = 1.0;
  color.b = 1.0;
}

Using this technique and experimenting with the values in the conditional helps us confirm that gl_FragCoord does indeed run from 0 to the size of the window in each dimension.

Visual Effects in Shaders

So after that quick introduction, let’s get to the heart of the matter, making visual effects using shaders. We are going to focus on 2D effects and are going to take inspiration from a blog post by John Nelson called Firefly Cartography. In it John describes a 3 step technique to make highly stylized thematic maps with dark moody backgrounds and a bright glowing data layer. He has some thoughts on what makes these maps visually appealing, and we are going to try and replicate (partially) that in Tangram. Our end result will look like this.

Zoomed In

Its not quite as polished as John Nelson’s Firefly maps, but it seems a decent start.

Step 1: The desaturated base map

The first step is creating the underlying desaturated base map to show the land. For this we will use a raster based terrain layer created by Stamen, and run image processing on it in real time a shader.

To import the terrain layer we add a new source

sources:
  stamen-terrain:
    type: Raster
    url: http://a.tile.stamen.com/terrain-background/{z}/{x}/{y}.jpg

The terrain layer actually looks like this.

To darken it we apply the following shader code

color: |
  // Compute luminance, via https://mapzen.com/documentation/tangram/Raster-Overview/
  float luma = dot(color.rgb, vec3(0.299, 0.587, 0.114));
  // Desaturate pixel.
  color.rgb = vec3(luma);

  // Adjust brightness and contrast
  float contrast = 1.8;
  float brightness = -0.8 * luma;
  color.rgb = contrast * (color.rgb - .5) + .5 + brightness;

Step 2: Masked highlight area and vignette

This part doesn’t use shaders, so I won’t go into much details (also I skipped the vignette ? ). In brief I loaded the country polygons we used in the last blog post and drew them on top of the basemap. I then used a filter to to make the polygon for the USA transparent while leaving all others a dark translucent black. If you are trying to do something like this, make sure to set the blend-mode of this layer to inlay .

Step 3: Single bright, glowing thematic layer.

I didn’t have any data in mind for this experiment, so we will plot the cities and populated places that come from OpenStreetMap. We’ll also adjust the size of the dots to roughly represent population. Lets see what the plain, pre-shader, dots look like.

Not bad, but not glowing. Here is the code for our shader to add simple glow effect. I did my best to follow the description of how to create a glow from John’s article.

citydot:
    base: points    
    blend: inlay
    shaders:
      blocks:
        color: |
          // We are going to use texture coordinates to color our dots.
          // What does texcoord look like?
          //
          // (0,1)----(1,1)
          //   |--------|
          //   |--------|
          //   |--------|
          // (0,0)----(1,0)
          //
          // v_texcoord is a _varying_ set by tangram.
          // see https://github.com/tangrams/tangram/blob/master/src/styles/points/points_fragment.glsl

          // Some coordinate checking debug code. Use this to get a sense of
          // the bounds of texcood space
          // vec2 uv = v_texcoord;
          // if(uv.x > 0.9 && uv.y > 0.9) {
          //   color.r = 1.0;
          //   color.g = 1.0;
          //   color.b = 1.0;
          // }

          // Now lets draw a circle. (and make it glow)

          // This is how distance to the center is computed in
          // the default point shader.
          vec2 uvd = v_texcoord * 2. - 1.;
          float point_dist = length(uvd);

          float centerRingRadius = 0.20;
          // Set color and alpha
          if (point_dist < centerRingRadius) {
            // Inside the center ring, use pure white
            color.rgba = vec4(1., 1., 1., 1.);
          } else {
            // Outside the center ring, don't modify the color set in the style
            // but change the alpha so that it is highest near the center and 0
            // at the edge.
            color.a = (1. - point_dist);
          }

Lets discuss the shader a bit. We are running the shader each piece of geometry in this layer, that is, for each circle that is visible. The shader will run for each pixel in each circle. To find our where the current pixel is we use v_texcoord . This is a coordinate in texture space. A texture is how images are manipulated in WebGL, and you can imagine the texture space as being the space of a square image covering the bounding box of our circle. v_texcoord is a variable created by Tangram itself with the coordinate for the current pixel in that texture space for this geometry.

To create our glowing circle we set the alpha of the pixel based on the distance from the center of the texture space, closer points are opaque and further points are transparent, resulting in a circle. We also make a small opaque white circle in the middle of it all.

And there it is! Budget firefly cartography—on the fly! Note that this is a fully interactive slippy map, and as you pan and zoom, different cities will show up based on the zoom level.

What about that mouse?

So one appeal of doing effects like this in this way is that they run in real time, and while the map above demonstrates that by being pan-able and zoom-able we could push this forward a bit by exploring animation and interaction. Lets take a small peek into what’s involved in doing this.

Lets add some code so that we can mouse over one of those place markers and have them change color and pulse a bit. To have them change color there are three important things we need to figure out:

• How to tell the shader where the mouse is?
• How to figure out which dots are close to the mouse?
• How to change change the color based on that informtion?

To do the first we are going to have to put on our uniforms. A uniform is a way to describe a variable that we can give to the shader program once its running. It’s called a uniform because it will have the same value for each pixel/fragment that the shader is processing. To do this in Tangram we add a new block to our shader style.

styles:
  citydot:
    base: points
    blend: inlay
    shaders:
      uniforms:
        u_mouse_x: 0
        u_mouse_y: 0

The uniforms u_mouse_x and u_mouse_y are where we are going to store the mouse coordinates for the shader, we can pass in those coordinates from our JavaScript code with the following snippet.

var layer = Tangram.leafletLayer({
  scene: 'scene.yaml',
});
layer.addTo(map);
var scene = layer.scene;

var requestRedraw = function() {
  window.requestAnimationFrame(function() {
    scene.requestRedraw();
  });
}

// Interaction callbacks
layer.setSelectionEvents({
  hover: function(selection) {
    if (selection.pixel && selection.leaflet_event.target) {
      var x = selection.pixel.x;
      var y = selection.pixel.y;

      // We update the uniform we declared in the yaml scene file
      // directly and then redraw.
      scene.styles.citydot.shaders.uniforms.u_mouse_x = x;
      scene.styles.citydot.shaders.uniforms.u_mouse_y = y;
      requestRedraw();
    }
  },
});

Next we need to calculate how close a point is to the mouse. This is something we would do inside our shader

# The fragment shader
color: |
  // read the mouse coordinate uniforms and normalize into 0-1 range
  // Check if the mouse if near this fragment
  //
  // We need to convert mouse position from css pixels to device pixels
  // for the calculations to work on 'retina' screens as gl_FragCoord is in
  // device pixels.
  vec2 mousePos = vec2(u_mouse_x, u_mouse_y) / u_resolution * u_device_pixel_ratio;
  // Flip the y axis to match this coordinate system
  mousePos.y = 1. - mousePos.y;
  // Calculate the normalized position of this fragment/pixel
  vec2 fragmentPos = gl_FragCoord.xy / u_resolution;

  // Find the distance to the mouse
  float mouseDistance = abs(distance(mousePos, fragmentPos));

Then we can set the pixel color based on distance to the mouse.

if (mouseDistance < 0.045) {
  color.rgb = vec3(1.00,0.39,0.28);
}

// Set color and alpha
if (point_dist < centerRingRadius) {
  color.rgba = vec4(1., 1., 1., 1.);
} else {
  color.a = (1. - point_dist);
}

To make it pulse we need to adjust the size of the whole thing over time. We’ll use a shaping function based on the sin function to do this. Instead of our previous block of code, we can use the following to achieve this.

float pulse = 0.0
if (mouseDistance < 0.045) {
  color.rgb = vec3(1.00,0.39,0.28); //orange
  // u_time is a uniform provided by Tangram that represents how much time
  // has passed since the scene started rendering. We will turn that into an
  // oscillation using the sin function with some offsets to control its size.
  pulse = sin(u_time * 1.45) * 0.5 + 0.5;
  pulse = clamp(pulse, 0., 0.3);
}

// Set color and alpha
// we use our pulse value to grow and shrink the whole dot over time.
if (point_dist < centerRingRadius - (pulse * .1)) {
  color.rgba = vec4(1., 1., 1., 1.);
} else {
  color.a = (1. - point_dist - pulse);
}

If you look carefully at the gif above you will see the orange dot pulsing slowly! You can also see this live here.

If you look closely you will notice that this is not a true ‘selection’. If we have the mouse in the right place, only part of the dot will be orange and glow. This is because we are doing this all one pixel at a time, without any real model of the overall geometry (we are in Lilliput remember). If I were trying to make a true selection I would use the Click API tangram provides combined with the techniques in this post (such as setting a uniform to store where we clicked) to achieve that, i’d also set the color in a more data driven way, similar to how the size of each point is set.

Whew! ?

There we are, an introduction to shader effects in Tangram. My observation is that shaders make traditional image processing techniques (such as saturation and brightness adjustment) quite straightforward. However more thought (and math) is needed to do things that are geometric in nature (though it should be remembered that modifying actual geometry is best done in a vertex shader which we haven’t covered here) . The power is there though, as can be seen if you take some time to browse advanced shader galleries, or even the advanced Tangram examples.

Another couple of resources I wanted to mention in closing are Tangram Blocks and stack.gl/glslify. These are both repositories of small chunks of shader code that provide common utilities you might find handy. As you may be able to guess from the name, Tangram Blocks provide for direct integration with Tangram based map.

Here is the link to a live version of the final result with full source code.