de-kAn-'str&kt

YouTube Smash Up

Introduction

Each week, we take the top 10 videos of YouTube and resynthesize the #1 video using the remaining 9 videos. We’ll continue doing so until one of our videos ends up in the top 10. The process, called “Smash Up” is a new kind of remix/mosaicing process which learns tiny perceptual fragments of audio and video using a computational model of audiovisual perception. If we were to train this model with a higher level of detail, with more objects from more videos, we could plausibly even resynthesize the number one YouTube video in a way that looked perceptually indistinguishable from the original. However, the process is not merely copying the file, but re-creating it out of entirely new material. By working with only the top 10 videos, the hope is to learn the appearance of YouTube’s most viral videos by creating a pseudo-’YouTube virus’ that feeds on viral videos. This work further tries to bring to focus how copyright laws will treat such works of art in the future, when our computational tools for remixing media are able to understand much more about the content rather than simply their textual meta-data (see this blog post expanding on the issue of copyright in my work in more detail).

How does it work?

Or to better phrase it as one of the viewers asked, “The hell kind of mashup is that?”. Fair question. It’s one based on a robot perception. Think of this as how Google might do a mashup. There are two stages, learning and synthesis. The learning phase looks at the top 2-10 videos and finds recurring patterns of light and sound. These patterns are well-studied audio and visual features that things like Google’s Search By Image may use in clustering similar content together. We find tiny chunks of audio and video that are unique across the database of videos and store these. These tiny chunks of video and sound create a knowledge-base of what the world looks and sounds like. After the robot model has an idea of what the most viral videos look and sound like, the second phase organizes all of this content into a new synthesis using the number one YouTube video. This organization, or interpretation, gives us the semblance of the original video, albeit an incredibly abstract one. We may be able to pick out movements of objects, even though our model only knows about proto-objects, or only understand noise, and that says a lot about our own perception.

Can I try it?

I’ve made a free iPhone/iPad app which you can get here: PhotoSynthesizer. It only does images, and is a very early prototype, but it will hopefully show what is possible with this kind of process.

What else has been done with this?

Some of the earlier work I’ve done can be found on my website such as a resynthesis of Jan Svankmajer’s work, a resynthesis of The Simpson’s intro using only the Family Guy, and a resynthesis of Michael Jackson’s Beat It using Nature Recordings.