- analytic cubism, appropriation, burroughs, collage, concatenative synthesis, copyright, dada, digital, frederic jameson, inference, intention, john oswald, law, michael jackson, mosaic, pastiche, perception, sampling, tristan tzara
The following article is written for the LUCID Studio for Speculative Art based in India.
Introduction
My work in audiovisual resynthesis aims to create models of how humans represent and attend to audiovisual scenes. Using pattern recognition of both audio and visual material, these models use large corpora of learned audiovisual material which can be matched to ongoing streams of incoming audio or visual material. The way audio and visual material is stored and segmented within the model is based heavily on neurobiology and behavioral evidence (the details are saved for another post). I have called the underlying model Audiovisual Content-based Information Description/Distortion (or ACID for short).
As an example, a live stream of audio may be matched to a database of learned sounds from recordings of nature, creating a re-synthesis of the audio environment at present using only pre-recorded material from nature itself. These learned sounds may be fragments of a bird chirping, or the sound of footsteps. Incoming sounds of someone talking may then be synthesized using the closest sounding material to that person talking, perhaps a bird chirp or a footstep. Instead of a live stream, one can also re-synthesize a pre-recorded stream. Consider using a database of nature recordings and, instead of the live-stream, now use a pre-existing recording of Michael Jackson. The following video demonstrates the output using Michael Jackson’s “Beat It”.
Everything you hear comes from nature recordings (by Chris Watson). Try to realize what elements of Michael Jackson’s original recording remain “meaningful”. The beat of the song is incredibly predominant (@ 33 seconds). As well, some aspects of the lyrics are present and heavily cross-modally present with the influence of the visual (e.g. @ 1:27), though no words are audible as the database contained no words. Also consider what meaningful information may be present in the opposite scenario, i.e. using a database of Michael Jackson, and re-synthesizing nature sounds.
As a side note, I have developed a similar approach for visual resynthesis, taking segments of visual objects as the basis of the resynthesis algorithm, rather than segments of audio. The example below demonstrates a resynthesis of the introduction to The Simpsons using only material from the introduction to The Family Guy:
More examples are on my vimeo channel.
Audio Collage
The idea of audio collage is not new. Electronic musicians have investigated the technique within the practice of music concrète. The advent of digital sampling with devices such as the Fairlight CMI as well made the practice much more accessible. Plunderphonics, a technique by John Oswald in which he manually chopped and resynthesized a number of copyrighted albums, created entirely new landscapes and sounds of material. It was also formalized in his essay, “Plunderphonics, or Audio Piracy as a Compositional Prerogative” where he questioned where copyright can begin to claim ownership. He famously made use of Michael Jackson recordings after 12 years of developing the technique in his EP Plunderphonics, perhaps the most extensive use of sampling to date, which featured an image of Michael Jackson on a naked woman on the album cover.
The album itself made use of material from a variety of artists, all credited, such as Count Basie, Dolly Parton, Beethoven, and Michael Jackson to recreate unheard of sounds reminiscent of Stravinsky and The Beatles. Oswald believed that by not selling the album, he was not infringing on anyone’s copyright. However, he faced legal pressure by Jackson’s attorneys and was forced to stop releasing the album by CBS and Jackson’s attorneys, destroying all remaining prints of the album. Other artists such as Negativland which experienced similar legal battles with U2 and Cassetteboy whose audiovisual video collage depicting BBC material of Queen Elizabeth was stripped from Youtube are also of note. Numerous documentaries including RiP: A Remix Manifesto and Good Copy Bad Copy, books such as Cutting Across Media: Appropriation Art, Interventionist Collage, and Copyright Law and Lessig’s Remix, and funded studies such as Recut, Reframe, Recycle have also focused on the topic (Thanks to Nathan Harmer for additional links).
Perception is Inference
I have extended these questions into the very nature of perception, claiming the computational models I employ are plausible models of our own psychological modeling of audio and vision itself. I try to make explicit one possible mechanism of perception as a meaning making inference machine, an idea which dates back at least to Helmholtz in 1869. The very nature of inference entails understanding requires prior experiences, prior models, a set of known examples. By taking the small fragments of sound and rearranging them, these perceptual units lose their context, and necessarily their original meaning, and are only bounded together by the ongoing environment to create a new meaning based on organization of the existing environment. In other words, without the environment to make a sound, there is no synthesis of a sound. The meaning therefore is created both by the viewer, and the target of the synthesis. The fragments within the database have no intentionality or meaning attributed to it until it is re-organized, resynthesized, and re-appropriated within a new context. In the video above, this context is Michael Jackson’s Beat-It; however, not a single fragment of Michael Jackson’s Beat It appears in the audio output.
No Infringement of Copyright Intended
Where then does copyright hold stake within the computational models I have created? The ongoing environment provides the intentional influence of how the sounds are to be rearranged. If Michael Jackson were to appear in the environment and sing a song that also appears in the corpus, it is likely the synthesis would re-create Michael Jackson’s song. As our own mental machinery encompasses having heard Michael Jackson before, we are able to recognize Michael Jackson. However, now consider a database containing tiny fragments of Michael Jackson’s recordings and a target of birds chirping and taxis honking. Then, the only semblance of Michael Jackson is based on the tiny fragments that appear, and the organization of a song of Michael Jackson is no longer present.
Now the question appears, “Does copyright hold stake over the ongoing environment’s intentions?”, requiring no one to perform Michael Jackson for fear of the copyrighted re-synthesis? Or does copyright instead hold stake in the subtle fragments of sounds which were sampled from a copyrighted track? Let us say I set up this computational model as an installation environment where the database contains both Michael Jackson songs and nature recordings. As a target for synthesis, I have a microphone feeding live audio to the computational model. Until the microphone hears something *like* Michael Jackson, creating a synthesis using the tiniest fragment of a Michael Jackson recording, it seems I will not have violated copyright.
Even still, how could we have understood that a tiny fragment of a resynthesis was copyrighted in the first place? We will have had to matched every tiny fragment within our own perceptual machinery (i.e. recognition) to have understood that I had heard this fragment within a different context, that is, a copyrighted one. Though, is it not the case that any meaning making inference will necessarily be matched to a prior experience? If so, then isn’t copyright claiming our own experiences as copyright? Where does the context of sound take place within copyright? Could I not listen to the blow of trees and be reminded of Michael Jackson? Or more likely, hear the wind and be reminded of a Chris Watson recording?
The question I am getting at is “How does the meaning elicited by the new resynthesis change our notion of copyright?” Consider the opposite scenario, where Michael Jackson’s songs are no longer in the database, and instead we have nature recordings, as the video above does. What does copyright have to say for using the organization of Michael Jackson’s songs though using entirely different content? In this case it seems I am using the artist’s full intention, though not breaking any copyright as the material is not evidently materialized in the resynthesis.
Early 20th Century
The early Dada movement of the 1920′s also looked at resynthesis within a narrative context. During a surrealist rally in the 1920′s, Tristan Tzara is famously noted as standing in front of a theater and pulling random fragments of text out of a hat before the riot ensued and wrecked the theater. T.S. Eliot’s The Waste Land and John Dos Passos’ U.S.A. trilogy are also early examples of the cut-up technique popularized by Tzara. The technique was further made famous by Brion Gysin and William Burroughs during the 1950′s in their poetry, which heavily made use of the cut-up technique in a variety of fashions. Gysin mistakenly came across the technique as he made use of layers of newspapers while cutting paper on top of them using a razor blade (originally to protect the table). He noticed the cut-up fragments of newspaper created a juxtaposition of image and text that were strangely coherent and meaningful. The two key terms to stress are ‘coherent’ and ‘meaningful’, of which our perceptual systems cannot help but create from the world. In fact, a number of theories of neuro-cognitive behavior are also based on these premises. Interested readers are encouraged to read Ronald Rensink’s work on theorizing the phenomena of visual Change Blindness into “Coherence Theory”, and Shihab Shamma’s work on modeling auditory perception within a “Temporal Coherence Theory”.
Frederic Jameson also discusses the nature of collage, an early analytic cubist practice developed by Picasso and Braque which graced the start of the century, as a process which creates a new meaning from existing material. In contrast to pastiche or bricolage, collage seeks a new meaning, whereas pastiche is often used pejoratively to denote a random intention or imitation of existing intentions, e.g. 16th century forgeries/imitations.
Conclusion
The distinction of collage versus pastiche seems incredibly relevant to practitioners making use of sampling and collage. However, I doubt such a distinction would help anyone in court. Though it is curious to think of the opposing scenario, where the material content is not copyright, though the intention or organization of the content is. How does a digital future handle this distinction without referring to intentions of an artist? Further, how could it prove an artist’s intentions in the first place?
Related: Copyright Violation Notice from “Rightster”, YouTube’s “Copyright School”, EFF Wins Renewal of Smartphone Jailbreaking Rights Plus New Legal Protections for Video Remixing
