de-kAn-'str&kt

• artist spaces, arts, creative, culture, design, diy, experimental, hack space, india, media, organization

Jaaga – Creative Common Ground
Bangalore
http://www.jaaga.in/

CEMA – Center for Experimental Media Arts at Srishti School of Art, Design and Technology
Bangalore
http://cema.srishti.ac.in/site/

Bar1 – non-profit exchange programme by artists for artists to foster the local, Indian and international mutual exchange of ideas and experiences through guest residencies in Bangalore
Bangalore
http://www.bar1.org

Sarai – a space for research, practice, and conservation about the contemporary media and urban constellations.
New Dehli
http://www.sarai.net/

Khoj/International Artists’ Association – artist led, alternative forum for experimentation and international exchange
New Dehli
http://www.khojworkshop.org/

Periferry – To create a nomadic space for hybrid art practices. It is a laboratory for people cross- disciplinary practices. The project focuses on the creation of a network space for negotiating the challenge of contemporary cultural production. It is located on a ferry barge on river Brahmaputra and is docked in Guwhati, Assam.
Narikolbari, Guwahati
http://www.periferry.in/

Point of View – non-profit organization that brings the points of view of women into community, social, cultural and public domains through media, art and culture.
Bombay
http://www.pointofview.org/

Majilis – a center for rights discourse and inter-disciplinary arts initiatives
Bombay
http://majlisbombay.org/

Camp – not an “artists collective” but a space, in which ideas and energies gather, and become interests and forms.
Bombay
http://www.camputer.org/

ChitraKarKhana – A fully independent, small scale unit for experimental media.
Bombay
http://chitrakarkhana.net/

• No comments yet
• Permanent Link

• appearance modeling, computer vision, eye movements, face recognition, face tracking, opencv, video, vision

I’ve been working on developing a method for automatic head-pose tracking, and along the way have come to model facial appearances. I start by initializing a facial bounding box using the Viola-Jones detector, a well known and robust detector used for training objects. This allows me to centralize the face. Once I know where the 2D plane of the face is in an image, I can register an Active Shape Model like so:

After multiple views of the possible appearance variations of my face, including slight rotations, I construct an appearance model.

The idea I am working with is using the first components of variations of this appearance model for determining pose. Here I show the first two basis vectors and the images they reconstruct:

As you may notice, these two basis vectors very neatly encode rotation. By looking at the eigenvalues of the model, you can also interpret pose.

• No comments yet
• Permanent Link

• accelerate, apple, audio, audio processing, fft, iphone, osx, source code

Using the libraries pkmFFT and pkm::Mat, you can very easily perform a highly optimized short time fourier transform (STFT) with direct access to a floating-point based object.

Get the code on my github:
http://github.com/pkmital/pkmFFT
Depends also on: http://github.com/pkmital/pkmMatrix

/*
 *  pkmSTFT.h
 *
 *  STFT implementation making use of Apple's Accelerate Framework (pkmFFT)
 *
 *  Created by Parag K. Mital - http://pkmital.com
 *  Contact: parag@pkmital.com
 *
 *  Copyright 2011 Parag K. Mital. All rights reserved.
 *
 *	Permission is hereby granted, free of charge, to any person
 *	obtaining a copy of this software and associated documentation
 *	files (the "Software"), to deal in the Software without
 *	restriction, including without limitation the rights to use,
 *	copy, modify, merge, publish, distribute, sublicense, and/or sell
 *	copies of the Software, and to permit persons to whom the
 *	Software is furnished to do so, subject to the following
 *	conditions:
 *
 *	The above copyright notice and this permission notice shall be
 *	included in all copies or substantial portions of the Software.
 *
 *	THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
 *	EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
 *	OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
 *	NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
 *	HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
 *	WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
 *	FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
 *	OTHER DEALINGS IN THE SOFTWARE.
 *
 *
 *  Usage:
 *
 *  // be sure to either use malloc or __attribute__ ((aligned (16))
 *  size_t buffer_size = 4096;
 *  float *sample_data = (float *) malloc (sizeof(float) * buffer_size);
 *  pkm::Mat magnitude_matrix, phase_matrix;
 *
 *  pkmSTFT *stft;
 *  stft = new pkmSTFT(512);
 *  stft.STFT(sample_data, buffer_size, magnitude_matrix, phase_matrix);
 *  fft.ISTFT(sample_data, buffer_size, magnitude_matrix, phase_matrix);
 *  delete stft;
 *
 */

#include "pkmFFT.h"
#include "pkmMatrix.h"

class pkmSTFT
{
public:

	pkmSTFT(size_t size)
	{
		fftSize = size;
		numFFTs = 0;
		fftBins = fftSize/2;
		hopSize = fftSize/4;
		windowSize = fftSize;
		bufferSize = 0;

		initializeFFTParameters(fftSize, windowSize, hopSize);
	}

	void initializeFFTParameters(size_t _fftSize, size_t _windowSize, size_t _hopSize)
	{
		fftSize = _fftSize;
		hopSize = _hopSize;
		windowSize = _windowSize;

		// fft constructor
		FFT = new pkmFFT(fftSize);
	}

	void STFT(float *buf, size_t bufSize, pkm::Mat &M_magnitudes, pkm::Mat &M_phases)
	{
		// pad input buffer
		int padding = ceilf((float)bufSize/(float)fftSize) * fftSize - bufSize;
		float *padBuf;
		if (padding) {
			printf("Padding %d sample buffer with %d samples\n", bufSize, padding);
			padBufferSize = bufSize + padding;
			padBuf = (float *)malloc(sizeof(float)*padBufferSize);
			// set padding to 0
			memset(&(padBuf[bufSize]), 0, sizeof(float)*padding);
			// copy original buffer into padded one
			memcpy(padBuf, buf, sizeof(float)*bufSize);	}
		else {
			padBuf = buf;
			padBufferSize = bufSize;
		}

		// create output fft matrix
		numWindows = (padBufferSize - fftSize)/hopSize + 1;

		if (M_magnitudes.rows != numWindows && M_magnitudes.cols != fftBins) {
			printf("Allocating %d bins x %d windows matrix for STFT\n", fftBins, numWindows);
			M_magnitudes.reset(numWindows, fftBins, true);
			M_phases.reset(numWindows, fftBins, true);
		}

		// stft
		for (size_t i = 0; i < numWindows; i++) {

			// get current col of freq mat
			float *magnitudes = M_magnitudes.row(i);
			float *phases = M_phases.row(i);
			float *buffer = padBuf + i*hopSize;

			FFT->forward(0, buffer, magnitudes, phases);	

		}
		// release padded buffer
		if (padding) {
			free(padBuf);
		}
	}

	void ISTFT(float *buf, size_t bufSize, pkm::Mat &M_magnitudes, pkm::Mat &M_phases)
	{
		int padding = ceilf((float)bufSize/(float)fftSize) * fftSize - bufSize;
		float *padBuf;
		if (padding)
		{
			printf("Padding %d sample buffer with %d samples\n", bufSize, padding);
			padBufferSize = bufSize + padding;
			padBuf = (float *)calloc(padBufferSize, sizeof(float));
		}
		else {
			padBuf = buf;
			padBufferSize = bufSize;
		}

		pkm::Mat M_istft(padBufferSize, 1, padBuf, false);

		for(size_t i = 0; i < numWindows; i++)
		{
			float *buffer = padBuf + i*hopSize;
			float *magnitudes = M_magnitudes.row(i);
			float *phases = M_phases.row(i);

			FFT->inverse(0, buffer, magnitudes, phases);
		}

		memcpy(buf, padBuf, sizeof(float)*bufSize);
		// release padded buffer
		if (padding) {
			free(padBuf);
		}
	}

private:

	pkmFFT				*FFT;

	size_t				sampleRate,
						numFFTs,
						fftSize,
						fftBins,
						hopSize,
						bufferSize,
						padBufferSize,
						windowSize,
						numWindows;
};

• 2 comments
• Permanent Link

• accelerate, apple, audio, audio processing, fft, ifft, iphone, optimization, osx, source code

Apple’s Accelerate Framework can really speed up your code without thinking too much. And it will also run on an iPhone. Even still, I did bang my head a few times trying to get a straightforward Real FFT and IFFT working, even after consulting the Accelerate documentation (reference and source code), stackoverflow (here and here), and an existing implementation (thanks to Chris Kiefer and Mick Grierson). Still, the previously mentioned examples weren’t very clear as they did not handle the case of overlapping FFTs which I was doing in the case of a STFT or they did not recover the power spectrum, or they just didn’t work for me (lots of blaring noise).

Get the code on my github:
http://github.com/pkmital/pkmFFT

/*
 *  pkmFFT.h
 *
 *  Real FFT wraper for Apple's Accelerate Framework
 *
 *  Created by Parag K. Mital - http://pkmital.com
 *  Contact: parag@pkmital.com
 *
 *  Copyright 2011 Parag K. Mital. All rights reserved.
 *
 *	Permission is hereby granted, free of charge, to any person
 *	obtaining a copy of this software and associated documentation
 *	files (the "Software"), to deal in the Software without
 *	restriction, including without limitation the rights to use,
 *	copy, modify, merge, publish, distribute, sublicense, and/or sell
 *	copies of the Software, and to permit persons to whom the
 *	Software is furnished to do so, subject to the following
 *	conditions:
 *
 *	The above copyright notice and this permission notice shall be
 *	included in all copies or substantial portions of the Software.
 *
 *	THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
 *	EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
 *	OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
 *	NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
 *	HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
 *	WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
 *	FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
 *	OTHER DEALINGS IN THE SOFTWARE.
 *
 *  Additional resources:
 *      http://developer.apple.com/library/ios/#documentation/Accelerate/Reference/vDSPRef/Reference/reference.html
 *      http://developer.apple.com/library/ios/#documentation/Performance/Conceptual/vDSP_Programming_Guide/SampleCode/SampleCode.html
 *      http://stackoverflow.com/questions/3398753/using-the-apple-fft-and-accelerate-framework
 *      http://stackoverflow.com/questions/1964955/audio-file-fft-in-an-os-x-environment
 *
 *
 *  This code is a very simple interface for Accelerate's fft/ifft code.
 *  It was built out of hacking Maximilian (Mick Grierson and Chris Kiefer) and
 *  the above mentioned resources for performing a windowed FFT which could
 *  be used underneath of an STFT implementation
 *
 *  Usage:
 *
 *  // be sure to either use malloc or __attribute__ ((aligned (16))
 *  float *sample_data = (float *) malloc (sizeof(float) * 4096);
 *  float *allocated_magnitude_buffer =  (float *) malloc (sizeof(float) * 2048);
 *  float *allocated_phase_buffer =  (float *) malloc (sizeof(float) * 2048);
 *
 *  pkmFFT *fft;
 *  fft = new pkmFFT(4096);
 *  fft.forward(0, sample_data, allocated_magnitude_buffer, allocated_phase_buffer);
 *  fft.inverse(0, sample_data, allocated_magnitude_buffer, allocated_phase_buffer);
 *  delete fft;
 *
 */

#include <Accelerate/Accelerate.h>

class pkmFFT
{
public:

	pkmFFT(int size = 4096, int window_size = 4096)
	{
		fftSize = size;					// sample size
		fftSizeOver2 = fftSize/2;
		log2n = log2f(fftSize);			// bins
		log2nOver2 = log2n/2;

		in_real = (float *) malloc(fftSize * sizeof(float));
		out_real = (float *) malloc(fftSize * sizeof(float));
		split_data.realp = (float *) malloc(fftSizeOver2 * sizeof(float));
		split_data.imagp = (float *) malloc(fftSizeOver2 * sizeof(float));

		windowSize = window_size;
		window = (float *) malloc(sizeof(float) * windowSize);
		memset(window, 0, sizeof(float) * windowSize);
		vDSP_hann_window(window, window_size, vDSP_HANN_DENORM);

		scale = 1.0f/(float)(4.0f*fftSize);

		// allocate the fft object once
		fftSetup = vDSP_create_fftsetup(log2n, FFT_RADIX2);
		if (fftSetup == NULL) {
			printf("\nFFT_Setup failed to allocate enough memory.\n");
		}
	}
	~pkmFFT()
	{
		free(in_real);
		free(out_real);
		free(split_data.realp);
		free(split_data.imagp);

		vDSP_destroy_fftsetup(fftSetup);
	}

	void forward(int start,
				 float *buffer,
				 float *magnitude,
				 float *phase)
	{
		//multiply by window
		vDSP_vmul(buffer, 1, window, 1, in_real, 1, fftSize);

		//convert to split complex format with evens in real and odds in imag
		vDSP_ctoz((COMPLEX *) in_real, 2, &split_data, 1, fftSizeOver2);

		//calc fft
		vDSP_fft_zrip(fftSetup, &split_data, 1, log2n, FFT_FORWARD);

		split_data.imagp[0] = 0.0;

		for (i = 0; i < fftSizeOver2; i++)
		{
			//compute power
			float power = split_data.realp[i]*split_data.realp[i] +
							split_data.imagp[i]*split_data.imagp[i];

			//compute magnitude and phase
			magnitude[i] = sqrtf(power);
			phase[i] = atan2f(split_data.imagp[i], split_data.realp[i]);
		}
	}

	void inverse(int start,
				 float *buffer,
				 float *magnitude,
				 float *phase,
				 bool dowindow = true)
	{
		float *real_p = split_data.realp, *imag_p = split_data.imagp;
		for (i = 0; i < fftSizeOver2; i++) {
			*real_p++ = magnitude[i] * cosf(phase[i]);
			*imag_p++ = magnitude[i] * sinf(phase[i]);
		}

		vDSP_fft_zrip(fftSetup, &split_data, 1, log2n, FFT_INVERSE);
		vDSP_ztoc(&split_data, 1, (COMPLEX*) out_real, 2, fftSizeOver2);

		vDSP_vsmul(out_real, 1, &scale, out_real, 1, fftSize);

		// multiply by window w/ overlap-add
		if (dowindow) {
			float *p = buffer + start;
			for (i = 0; i < fftSize; i++) {
				*p++ += out_real[i] * window[i];
			}
		}
	}

private:

	size_t				fftSize,
						fftSizeOver2,
						log2n,
						log2nOver2,
						windowSize,
						i;					

	float				*in_real,
						*out_real,
						*window;

	float				scale;

    FFTSetup			fftSetup;
    COMPLEX_SPLIT		split_data;

};

• 16 comments
• Permanent Link

• acoustic, augmented, behavior, ecology, embodied, enactive, gesture, liminality, memory, perception, reality, sensors, situated, sonic, sound field, soundscapes

I recently gave two talks, one for the PhDs based in the Electronic Music Studios, and another for the PhDs in Arts and Computational Technology. I received some very valuable feedback, and having to incorporate what I’ve been working on in a somewhat presentable manner also had a lot of benefit. The talk abstract (which is very abstract) is posted below with a few references listed. Please feel free to comment and open a discussion, or post any references that may be of interest.

Abstract:
An augmented sonic reality aims to register digital sound content with an existing physical space. Perceptual mappings between an agent in such an environment and the augmented content should be both continuous and effective, meaning the intentions of an agent should be taken into consideration in any affective augmentations. How can an embedded intelligence such as an iPhone equipped with detailed sensor information such as microphone, accelerometer, gyrometer, and GPS readings infer the behaviors of its user in creating affective, realistic, and perceivable augmented sonic realities tied to their situated experiences? Further, what can this augmented domain reveal about our own ongoing sensory experience of our sonic environment?

Keywords: augmented, reality, sonic, enactive, perception, memory, behavior, sensors, gesture, embodied, situated, acoustic, ecology, liminality

References:
Augoyard and Torgue, “Sonic Experience”, McGill-Queen’s University Press, 2005.
Arfib, D. “Organised Sound”, 2002.
E. Corteel, “Synthesis of directional sources using Wave Field Synthesis, possibilities and limitations.” EURASIP Journal on Advances in Signal Processing, special issue on Spatial Sound and Virtual Acoustics, January, 2007
Lemaitre, G., Houix, O., Visell, Y., Franinovic, K., Misdariis, N., Susini, P. “Toward the Design and Evaluation of Continuous Sound in Tangible Interfaces: The Spinotron”, International Journal of Human Computer Studies, no 67, 2009
K. Nguyen, C. Suied, I. Viaud-Delmon, O. Warusfel, “Spatial audition in a static virtual environment : the role of auditory-visual interaction.” Journal of Virtual Reality and Broadcasting, 2009
M. Noisternig, B. Katz, S. Siltanen, L. Savioja, “Framework for Real-Time Auralization in Architectural Acoustics.” Acta acustica united with Acustica, vol. 94, no 6, November, 2008
R. Murray Schafer, “The Soundscape”, Destiny Books, 1977.
J. Tardieu, P. Susini, F. Poisson, P. Lazareff, S. McAdams, “Perceptual study of soundscapes in train stations.” Applied Acoustics, vol. 69, no 12, December, 2008
Strategies of mapping between gesture data and synthesis model parameters using perceptual spaces. D. Arfib, J. M. Couturier, L. Kessous, V. Verfaille. Organised Sound, International Journal of Music Technology, Volume 7, Issue 2 , pages 127-144, 2002.
D. Arfib, J-M. Couturier, L. Kessous , “Expressiveness and digital musical instrument design”, in Journal of New Music Research,Vol. 34, No. 1, pages 125 – 136, 2005.
N. d’Alessandro, O. Babacan, B. Bozkurt, T. Dubuisson, A. Holzapfel, L. Kessous, A. Moinet, M. V. Lieghe, “RAMCESS 2.X framework – expressive voice analysis for realtime and accurate synthesis of singing”, Journal On Multimodal User Interfaces, Springer Berlin/Heidelberg, Vol. 2, Nr. 2, September, pages 133-144, 2008.
L. Kessous, G. Castellano, G. Caridakis, Multimodal emotion recognition in speech-based interaction using facial expression, body gesture and acoustic analysis, Journal on Multimodal User Interfaces, Vol. 3, Issue 1, Springer Berlin/Heidelberg, December 12, pages 33-48, 2009.
G.Caridakis, K. Karpouzis, M. Wallace, L. Kessous, N.Amir, Multimodal user’s affective state analysis in naturalistic interaction, Journal on Multimodal User Interfaces, Vol. 3, Issue 1, Springer Berlin/Heidelberg, December 15, pages 49-66, 2009.
Anton Batliner, Stefan Steidl, Bjoern Schuller, Dino Seppi, Turid Vogt, Johannes Wagner, Laurence Devillers, Laurence Vidrascu, Noam Amir, Loic Kessous and Vered Aharonson. Whodunnit – Searching for the Most Important Feature Types Signalling Emotion-Related User States in Speech. Computer Speech and Language, volume 25, No. 1, pages 4–28, 2011.

• No comments yet
• Permanent Link

• blog, david bordwell, diem, open-source, tim j smith, visual cognition, visualization

Tim J Smith, expert in scene perception and film cognition, and of The DIEM project [1] recently starred as a guest blogger for David Bordwell, a leading film theorist with an impressive list of books and publications widely used in film cognition/film art research/studies [2]. In his article featured on David’s site, Tim expands on his research on film cognition including continuity editing [3], attentional synchrony [4], and the project we worked on in 2008-2010 as part of The DIEM Project. Since Tim’s feature on David Bordwell’s blog, The DIEM Project saw a surge of publicity and our vimeo video loads going higher than 200,000 in a single day and features on dvice, slashfilm, gizmodo, Rogert Ebert’s facebook/twitter, and the front page of imbd.com.

Not to mention, our tools and visualizations are finally reaching an audience with interests in film, photography, and cognition. If you haven’t yet seen some of our videos, please head on over to our vimeo page, where you can see a range of videos embedded with eye-tracking of participants and many different visualizations of models of eye-movements using machine learning, or start by reading Tim’s post on DavidBordwell.net. You can also visit our website and create your own visualizations with our completely open-source tool, CARPE, and our completely royalty free database, the DIEM database. I’ve linked a few of my favorite videos below which were all made with CARPE with the last one showing a unique visualization of a movie as it’s motion:

Montage of 4 Visualizations of Eye-movements during Charlie Bit My Finger from TheDIEMProject on Vimeo.

tv the simpsons 860×528 web from TheDIEMProject on Vimeo.

Eye movments during the Video Republic from TheDIEMProject on Vimeo.

Eye Movements during a Movie Trailer of Ice Age 3 from TheDIEMProject on Vimeo.

Eye-movements during 50 People, 1 Question (Brooklyn) from TheDIEMProject on Vimeo.

[1]. The DIEM Project was funded by the Leverhulme Trust (Grant Ref F/00-158/BZ) and the ESRC (RES 062-23-1092) and awarded to John M Henderson.
[2]. http://en.wikipedia.org/wiki/David_Bordwell
[3]. http://www.era.lib.ed.ac.uk/bitstream/1842/1076/1/smith_ATOCE_0506.pdf
[4]. Mital, P.K., Smith, T. J., Hill, R. and Henderson, J. M., “Clustering of gaze during dynamic scene viewing is predicted by motion,” Cognitive Computation

• No comments yet
• Permanent Link

• art, captincaptin, cave, computer vision, exhibit, exhibition, immersive, installation, london, projection, sculpture, source code, tracking, video, video art

As part of a system of numerous dynamic connections and networks, we are reactive and deterministic to a complex system of cause and effect. The consequence of our actions upon our selves, the society we live in and the broader natural world is conditioned by how we perceive our involvement. The awareness of how we have impacted on a situation is often realised and processed subconsciously, the extent and scope of these actions can be far beyond our knowledge, our consideration, and importantly beyond our sensory reception. With this in mind, how can we associate our actions, many of which may be overlooked as customary, with for instance, the honey bee depopulation syndrome or the declining numbers of Siberian Tigers.

Responsive Ecologies is part of an ongoing collaboration with ZSL London Zoo and Musion Academy. Collectively we have been exploring innovative means of public engagement, to generate an awareness and understanding of nature and the effects of climate change. All of the contained footage has come from filming sessions within the Zoological Society; this coincidentally has raised some interesting questions on the spectacle of captivity, a issue which we have tried to reflect upon in the construction and presentation of this installation. The nature of interaction within Responsive Ecologies means that a visitor to the space can not simply view the installation but must become a part of its environment. When attempting to perceive the content within the space the visitor reshapes the installation. Everybody has a degree of impact whether directed or incidental, and when interacting as a group it is interesting to see how collective behaviour can develop and incite the outcome of the work.

captincaptin and Parag K Mital exhibited Responsive Ecologies at the Watermans between 6th December 2010 and the 21st January 2011. The installation was in the form of a 360 degrees multi-screened projection or CAVE (Cave Automatic Virtual Environment). Visitors to the exhibition would enter the CAVE through a passageway leading from the gallery entrance. All four sides of the CAVE were back projected with each side connecting to form a large continuous projection. The presence of people within the space would be tracked and used to deconstruct and interlace the video in response to their movement. The video documentation below was taken from the installation (throughout this video the camera is panning around the space in order to record all sides of the CAVE).

Responsive Ecologies from pkmital on Vimeo.

Source code is also available with an example part of the actual installation (10 second clip as the whole video is far too long) – though you may also need 4 monitors/projectors or scale down the size of the screen in the code (SCREEN_WIDTH and SCREEN_HEIGHT variables in testApp.h): http://github.com/pkmital/Responsive-Ecologies

More information:
http://captincaptin.co.uk
http://pkmital.com

• No comments yet
• Permanent Link

• cycling74, kinect, maxmsp, motion tracking, nite, opensource, osc, osx, primesense

This guide will help to get you running PrimeSense NITE’s Skeleton Tracking inside XCode on your OSX.  It will also help you stream that data in case you’d like to use it in another environment such as Max.  An example Max patch is also available.

PrimeSense NITE Skeletonization and Motion Capture to Max/MSP via OSC from pkmital on Vimeo.

Prerequisites:

0.) 1 Microsoft Kinect or other PrimeSense device.

1.) Install XCode and Java Developer Package located here: https://connect.apple.com/cgi-bin/WebObjects/MemberSite.woa/wa/getSoftware?bundleID=20719 – if you require a Mac OSX Developer account, just register at developer.apple.com since it is free.

2.) Install Macports: http://www.macports.org/

3.) Install libtool and libusb > 1.0.8:

$ sudo port install libusb-devel +universal

4.) Get the OpenNI Binaries for Mac OSX: http://www.openni.org/downloadfiles

5.) Install OpenNI by unzipping the file OpenNI-Bin-MacOSX (-v1.0.0.25 at the time of writing) and running,

$ sudo ./install.sh

6.) Get SensorKinect from avin2: https://github.com/avin2/SensorKinect/tree/unstable/Bin

7.) Install SensorKinect by unzipping and running

$ sudo ./install.sh

8.) Install OpenNI Compliant Middleware NITE from Primesense for Mac OSX: http://www.openni.org/downloadfiles

9.) Install NITE by unzipping and running

$ sudo ./install.sh

When prompted for a key, enter the key listed on the openni website.

Getting it up and running:

1.) Download the OSC example from here: https://github.com/pkmital/StickFigureOSC – you can still download the project without having git, just look for the “Downloads” link to the right of the screen.

2.) After downloading (and extracting if you downloaded the zip file), navigate to ./StickFigure and open the XCode Project file.

3.) Compile and run, and hopefully there are no problems…

Optional:
1.) Try and visualize the data in Max/MSP using the Max patch bundled inside the git repository mentioned in step 10: https://github.com/pkmital/StickFigureOSC/blob/master/kinect_skeleton.zip

With this, you can also do multiple person skeletonization and motion tracking, though if you are using the OSC information, you might want to include an OSC Tag for which person’s joint are being sent.

• 15 comments
• Permanent Link

• animals, art, captincaptin, cinema, film, immersive, installation, interactive, london, musion academy, watermans, zsl london zoo

Come checkout the Waterman’s Art Centre from the 6th of December until the 21st of January for an immersive and interactive visual experience entitled “Responsive Ecologies” developed in collaboration with artists captincaptin. We will also be giving a talk on the 10th of December from 7 p.m. – 9 p.m. during CINE: 3D Imaging in Art at the Watermans Center.

Responsive Ecologies is part of a wider ongoing collaboration between artists captincaptin, the ZSL London Zoo and Musion Academy. Collectively they have been exploring innovative means of public engagement, to generate an awareness and understanding of nature and the effects of climate change. All of the contained footage has come from filming sessions within the Zoological Society; this coincidentally has raised some interesting questions on the spectacle of captivity, a issue which we have tried to reflect upon in the construction and presentation of this installation. The nature of interaction within Responsive Ecologies means that a visitor to the space cannot simply view the installation but must become a part of its environment. When attempting to perceive the content within the space the visitor reshapes the installation. Everybody has a degree of impact whether directed or incidental, and when interacting as a group it is interesting to see how collective behaviour can develop and incite the outcome of the work.

The installation will encompass participants in 360 degrees with a small door opening. Some pictures of a mock setup using 2 projectors:

And a short video:

Responsive Ecologies Test from pkmital on Vimeo.

Using overhead motion capture and 4 channels of video projection, users are invited to interact in an immersive cinematic environment. The video above shows a demo proof-of-concept for the upcoming installation in Waterman’s Art Centre, London, UK, where artists captincaptin and Parag K Mital have been collaborating to install their piece, “Responsive Ecologies” for their residency the next two months. The work occurs as part of a larger on-going collaboration with ZSL London Zoo and the Musion Academy.

Earlier I posted a video showing how the motion detection works:

Overhead Blob Detection from pkmital on Vimeo.

Using OpenCV with openFrameworks to do blob detection for an upcoming installation entitled “Responsive Ecologies” at the Waterman’s Art Centre in London, UK in collaboration with artists captincaptin.

Full exhibition text follows:

As part of a system of numerous dynamic connections and networks, we are reactive and deterministic to a complex system of cause and effect. The consequence of our actions upon our selves, the society we live in and the broader natural world is conditioned by how we perceive our involvement. The awareness of how we have impacted on a situation is often realised and processed subconsciously, the extent and scope of these actions can be far beyond our knowledge, our consideration, and importantly beyond our sensory perception. With this in mind, how can we associate our actions, many of which may be overlooked as customary, with for instance, the honey bee depopulation syndrome or the declining numbers of Siberian Tigers?

The expanding distance between our lives and the natural world has created a detachment in which we begin to lose the emotion of responsibility. The exhibition references this human-nature dislocation, creating a dynamic environment which can be effeced and controlled by the audience. It aims to symbolise an embodiment of ecology, exploring the relations of organisms, and their interactions with the environment.

Responsive Ecologies is part of an ongoing collaboration with ZSL London Zoo and Musion Academy. Collectively we have been exploring innovative means of public engagement, to generate an awareness and understanding of nature and the effects of climate change. All of the contained footage has come from filming sessions within the Zoological Society; this coincidentally has raised some interesting questions on the spectacle of captivity, a issue which we have tried to reflect upon in the construction and presentation of this installation. The nature of interaction within Responsive Ecologies means that a visitor to the space can not simply view the installation but must become a part of its environment. When attempting to perceive the content within the space the visitor reshapes the installation. Everybody has a degree of impact whether directed or incidental, and when interacting as a group it is interesting to see how collective behaviour can develop and incite the outcome of the work.

Gareth Goodison and Jonathan Munro have been working collaboratively as captincaptin since 2007, creating installations and sculptures, which interact and respond to public presence, to question the role of audience participation in the display and creation of contemporary art. They have presented their work at various new media events including Future Everything Festival, Abandon Normal Devices, Leeds Expo, and the V&A. Parag K Mital is a computational visual artist and PhD student at Goldsmiths Computing Dept. investigating questions of liminality, perception, and attention in mixed reality environments.

The exhibition continues outside the theatre, presenting more outcomes of this collaboration, including the work by artists Madi Boyd and Kira Zhigalina. Special thanks to ZSL London Zoo and Musion Academy.

http://www.watermans.org.uk/exhibitions/exhibitions/future-exhibitions/responsive-ecologies.aspx

• No comments yet
• Permanent Link

• computer vision, eye movements, face tracking, faceapi, gaze tracking, head tracking, mario, mutual information, openframeworks, opentl

The following demo works with SeeingMachines FaceAPI in openFrameworks controlling a Mario avatar.  It also has some really poor gesture recognition (and learning but it’s not shown here), though a threshold on the rotation DOF would have produced better results for the simple task of looking up/down left/right gestures.

6DOF Head Tracking from pkmital on Vimeo.

interfacing seeingmachines faceapi with openFrameworks to control a 3D mario avatar

This is just with the non-commercial license. The full commercial license (~$3000?) gives you access to lip/mouth tracking and eye-brows, as well as much more flexibility in how to use their api with different/multiple cameras and accessing image data.

Of course, there are other initiatives at producing similar results. Mutual information based template trackers, for instance, seem to be state-of-art. Take a look at recent work by Panin and Knoll using OpenTL:

I imagine a lot of people would like this technology.

• No comments yet
• Permanent Link