Measurement Lab - Visualizing the Health of the Internet

New America

Processing, analyzing and visualizing 800,000,000+ speed tests.

See it Live

Challenge

Measurement Lab is a research partnership between New America’s Open Technology Institute, Google Open Source Research, Princeton University’s PlanetLab and others.

The project has collected over 800 million network diagnostic test results from around the globe since early 2009. These test results comprise the largest public dataset of global internet speeds.

The data set is publicly available in Google’s BigQuery, but its sheer size has made it difficult for interested parties to work with. As a consequence, this treasure trove of information, critical to the understanding of the health of the Internet, has remained out of reach for policy makers, researchers, and others who could benefit immensely from its insights.

M-Lab came to the Bocoup Data Visualization Team to make this data more accessible, transparent, and engaging for potential audiences.

Solution

Bocoup worked with the Measurement Lab team to create a comprehensive data transformation, storage, and visualization system.

We first designed and developed a scalable data pipeline using Google’s Dataflow, enabling us to access, augment and transform the data housed in Bigquery. Next, our team designed and developed a fast REST API built using Python Flask on top of Google’s Bigtable that enabled us to access the data through a variety of facets and filters.

This view into the data, enabled us to design and develop a full-stack application with rich visualizations that allows users to view internet measurement data in specific locations and timeframes, as well as investigate consumer and transit internet service providers. Our work has also exposed this data in a variety of formats to interested public parties through downloads and a public version of the above mentioned REST API.

Impact

Through our work together we have opened up access to Measurement Lab’s incredible collection of data. Researchers, policy makers, journalists and members of the public now have easy and direct access to investigate the health of the Internet at local and global levels. These tools enable investigations into the impact of the internet on public issues ranging from health to security and from global trade to foreign policy.

More work like this at Bocoup