Bocoup & PBS: Building and Scaling a Realtime Node.js Application

This summer, Bocoup worked with Mozilla and PBS to build an open source, real-time web application for the 2012 US Presidential election. When we began, PBS (specifically, the folks at News Hour) had already completed a client-side application called “Map Center“. Map Center enabled users to view US demographic data and experiment with electoral college vote distributions.

Bocoup was tasked with the job of extending this application into a “second screen” experience, where the 10,000 anticipated television viewers could receive contextualized information in real time. This work took place in public (you can find the code repository on GitHub) and was built on a completely open source stack that included Debian, Node.js and Redis.

Given the expected “shape” of the traffic (over 10,000 people connecting during the broadcast, many at its onset), we were particularly interested in scalability. After the application was feature complete, we focused on finding and removing any bottlenecks that would stand in the way.

I’d like to take a moment to acknowledge the hard work of the people we worked with. Ben Moskowitz at Mozilla provided excellent oversight at the inception of the project. Justin Myers, Vanessa Dennis, and Travis Daub showed great diligence and patience in their work evangelizing new technology at PBS and really bringing the project to life. We look forward to working together again soon!

This post is the introduction to a three-part series covering the stress testing phase of the project:

First is a postmortem that drills down into my test results and explains how each test informed our understanding of the application’s bottlenecks, available here). Next, I cover my methodology to help others perform similar testing on their own apps; you can find that one here). Finally, I discuss how I set up a production-ready server from scratch.