Parag Kumar Mital

I’ve been interested in exploring ways of navigating media archives. Typically, you may use iTunes and go from artist to artist, or have managed to tediously classify your collection into genres. Some may still even browse their music through a file browser, perhaps making sure the folders and filenames of their collection are descriptive of the artist, album, year, etc… Though what about how the content actually sounds?

Wouldn’t it be nice to hear all music which shares similar sounds, or similar phrases of sounds? Research in the last 10-15 years have developed methods precisely to solve this problem and fall under the umbrella term content-based information retrieval (CBIR) algorithms, or uncovering the relationships of an archive through the information within the content. For images, Google’s Search by Image is a great example which only recently became public. For audio, audioDB and ShaZam are good examples of discovering music through the way it sounds, or the content-based relationships of the audio itself. Though, each of these interfaces present a list of matches to a image or audio query, making exploring the content-based relationships of a specific set of material difficult.

The video above demonstrates interaction with a novel 3D browser of a collection of music by one artist, Daphne Oram. The sounds are grouped in 3D space based on the way they sound, clustering together similar sounding material. Each of the 3 axes describes a grouping of sound frequencies. So a timbre, or a texture of sound. The position of the sound along one of these axes means there is a lot of that group of frequencies present in the sound file.

Exploring her work in this browser really demonstrates the variety of sounds she achieved. It also makes exploring the collection really fun to use, as there is a fun visual form, and you also get to hear stuff right away.

The browser has also been built to be a real-time tool for creating new sounds. Mousing over any of the tiny boxes (representing parts of audio files in the collection) triggers the clip to play. Since similar sounding clips are grouped closer together, one can “perform” the collection along perceptually coherent axes by moving the mouse along any of the axes.