Measurement Lab - Visualizing the Health of the Internet

Measurement Lab is a research partnership between New America’s Open Technology Institute, Google Open Source Research, Princeton University’s PlanetLab and others.

The project has collected over 800 million network diagnostic test results from around the globe since early 2009. These test results comprise the largest public dataset of global internet speeds.

The data set is publicly available in Google’s BigQuery, but its sheer size has made it difficult for interested parties to work with. As a consequence, this treasure trove of information, critical to the understanding of the health of the Internet, has remained out of reach for policy makers, researchers, and others who could benefit immensely from its insights.

M-Lab came to the Bocoup Data Visualization Team to make this data more accessible, transparent, and engaging for potential audiences.

Bocoup worked with the Measurement Lab team to create a comprehensive data transformation, storage, and visualization system.

We first designed and developed a scalable data pipeline using Google’s Dataflow, enabling us to access, augment and transform the data housed in Bigquery. Next, our team designed and developed a fast REST API built using Python Flask on top of Google’s Bigtable that enabled us to access the data through a variety of facets and filters.

This view into the data, enabled us to design and develop a full-stack application with rich visualizations that allows users to view internet measurement data in specific locations and timeframes, as well as investigate consumer and transit internet service providers. Our work has also exposed this data in a variety of formats to interested public parties through downloads and a public version of the above mentioned REST API.