TerraBeats

Sonifying the world

Team Members

Anton Sivertsson, Cynthia Ho, Elena Goncharuk, Josef Grahn, Julia Krantz, Lisa Andersson Lopez, Robin Palmberg, Tim Lachmann

Challenges

ESA/SNSA Challenge, Musical City, KTH Innovation Challenge,

Description

The world is in an ever changing state and it is often hard to grasp the severity of its changes. By utilizing data driven methods, we have managed to associate sounds to the environmental changes of the earth. The multimodal experience creates for an awareness about where the future of the world is heading.

We set out to use deep learning to make computers make songs out of certain geolocational areas. But what we found, after iterating the process at several time steps, was that it was possible to get a feel of how the earth is affected by everything that is happening in our society. Since we wanted to have a pipeline with as little manipulation by humans as possible, we set up a rule set and let the computers, based on data, create the sonifications related to each place.

The tools we have managed to produce are possible to use for a wide range of changes in the environment, everything from natural disasters to malicious attacks can be tracked and sonified.

The sonifications can be heard here: https://drive.google.com/open?id=1YWftCA2lul6Ta31As8WdYPvL5y4tsfll

Tools and Data

To accomplish our goal, we used deep learning, TensorFlow, spectral analysis, multivariate statistics as well as other data driven methods to classify areas and extract, through image recognition, features from satellite imagery (provided by the SNSA) that the AI could use to create the sonifications. To turn the extracted data into sounds, we set up a framework for the AI to use in the musical coding language called "ChucK" (http://chuck.cs.princeton.edu/).