Latest News

Tim J Smith guest blogs for David Bordwell

Tim J Smith, expert in scene perception and film cognition, and of The DIEM project [1] recently starred as a guest blogger for David Bordwell, a leading film theorist with an impressive list of books and publications widely used in film cognition/film art research/studies [2]. In his article featured on David’s site, Tim expands on his research on film cognition including continuity editing [3], attentional synchrony [4], and the project we worked on in 2008-2010 as part of The DIEM Project. Since Tim’s feature on David Bordwell’s blog, The DIEM Project saw a surge of publicity and our vimeo video loads going higher than 200,000 in a single day and features on dvice, slashfilm, gizmodo, Rogert Ebert’s facebook/twitter, and the front page of imbd.com.

Not to mention, our tools and visualizations are finally reaching an audience with interests in film, photography, and cognition. If you haven’t yet seen some of our videos, please head on over to our vimeo page, where you can see a range of videos embedded with eye-tracking of participants and many different visualizations of models of eye-movements using machine learning, or start by reading Tim’s post on DavidBordwell.net. You can also visit our website and create your own visualizations with our completely open-source tool, CARPE, and our completely royalty free database, the DIEM database. I’ve linked a few of my favorite videos below which were all made with CARPE with the last one showing a unique visualization of a movie as it’s motion:

Montage of 4 Visualizations of Eye-movements during Charlie Bit My Finger from TheDIEMProject on Vimeo.

tv the simpsons 860×528 web from TheDIEMProject on Vimeo.

Eye movments during the Video Republic from TheDIEMProject on Vimeo.

Eye Movements during a Movie Trailer of Ice Age 3 from TheDIEMProject on Vimeo.

Eye-movements during 50 People, 1 Question (Brooklyn) from TheDIEMProject on Vimeo.

[1]. The DIEM Project was funded by the Leverhulme Trust (Grant Ref F/00-158/BZ) and the ESRC (RES 062-23-1092) and awarded to John M Henderson.
[2]. http://en.wikipedia.org/wiki/David_Bordwell
[3]. http://www.era.lib.ed.ac.uk/bitstream/1842/1076/1/smith_ATOCE_0506.pdf
[4]. Mital, P.K., Smith, T. J., Hill, R. and Henderson, J. M., “Clustering of gaze during dynamic scene viewing is predicted by motion,” Cognitive Computation

Responsive Ecologies Documentation

As part of a system of numerous dynamic connections and networks, we are reactive and deterministic to a complex system of cause and effect. The consequence of our actions upon our selves, the society we live in and the broader natural world is conditioned by how we perceive our involvement. The awareness of how we have impacted on a situation is often realised and processed subconsciously, the extent and scope of these actions can be far beyond our knowledge, our consideration, and importantly beyond our sensory reception. With this in mind, how can we associate our actions, many of which may be overlooked as customary, with for instance, the honey bee depopulation syndrome or the declining numbers of Siberian Tigers.

Responsive Ecologies is part of an ongoing collaboration with ZSL London Zoo and Musion Academy. Collectively we have been exploring innovative means of public engagement, to generate an awareness and understanding of nature and the effects of climate change. All of the contained footage has come from filming sessions within the Zoological Society; this coincidentally has raised some interesting questions on the spectacle of captivity, a issue which we have tried to reflect upon in the construction and presentation of this installation. The nature of interaction within Responsive Ecologies means that a visitor to the space can not simply view the installation but must become a part of its environment. When attempting to perceive the content within the space the visitor reshapes the installation. Everybody has a degree of impact whether directed or incidental, and when interacting as a group it is interesting to see how collective behaviour can develop and incite the outcome of the work.

captincaptin and Parag K Mital exhibited Responsive Ecologies at the Watermans between 6th December 2010 and the 21st January 2011. The installation was in the form of a 360 degrees multi-screened projection or CAVE (Cave Automatic Virtual Environment). Visitors to the exhibition would enter the CAVE through a passageway leading from the gallery entrance. All four sides of the CAVE were back projected with each side connecting to form a large continuous projection. The presence of people within the space would be tracked and used to deconstruct and interlace the video in response to their movement. The video documentation below was taken from the installation (throughout this video the camera is panning around the space in order to record all sides of the CAVE).

Responsive Ecologies from pkmital on Vimeo.

Source code is also available with an example part of the actual installation (10 second clip as the whole video is far too long) – though you may also need 4 monitors/projectors or scale down the size of the screen in the code (SCREEN_WIDTH and SCREEN_HEIGHT variables in testApp.h): http://github.com/pkmital/Responsive-Ecologies


More information:
http://captincaptin.co.uk
http://pkmital.com

Streaming Motion Capture Data from the Kinect using OSC on Mac OSX

This guide will help to get you running PrimeSense NITE’s Skeleton Tracking inside XCode on your OSX.  It will also help you stream that data in case you’d like to use it in another environment such as Max.  An example Max patch is also available.

PrimeSense NITE Skeletonization and Motion Capture to Max/MSP via OSC from pkmital on Vimeo.

Prerequisites:

0.) 1 Microsoft Kinect or other PrimeSense device.

1.) Install XCode and Java Developer Package located here: https://connect.apple.com/cgi-bin/WebObjects/MemberSite.woa/wa/getSoftware?bundleID=20719 – if you require a Mac OSX Developer account, just register at developer.apple.com since it is free.

2.) Install Macports: http://www.macports.org/

3.) Install libtool and libusb > 1.0.8:

$ sudo port install libusb-devel +universal

4.) Get the OpenNI Binaries for Mac OSX: http://www.openni.org/downloadfiles

5.) Install OpenNI by unzipping the file OpenNI-Bin-MacOSX (-v1.0.0.25 at the time of writing) and running,

$ sudo ./install.sh

6.) Get SensorKinect from avin2: https://github.com/avin2/SensorKinect/tree/unstable/Bin

7.) Install SensorKinect by unzipping and running

$ sudo ./install.sh

8.) Install OpenNI Compliant Middleware NITE from Primesense for Mac OSX: http://www.openni.org/downloadfiles

9.) Install NITE by unzipping and running

$ sudo ./install.sh

When prompted for a key, enter the key listed on the openni website.

Getting it up and running:

1.) Download the OSC example from here: https://github.com/pkmital/StickFigureOSC – you can still download the project without having git, just look for the “Downloads” link to the right of the screen.

2.) After downloading (and extracting if you downloaded the zip file), navigate to ./StickFigure and open the XCode Project file.

3.) Compile and run, and hopefully there are no problems…

Optional:
1.) Try and visualize the data in Max/MSP using the Max patch bundled inside the git repository mentioned in step 10: https://github.com/pkmital/StickFigureOSC/blob/master/kinect_skeleton.zip

With this, you can also do multiple person skeletonization and motion tracking, though if you are using the OSC information, you might want to include an OSC Tag for which person’s joint are being sent.

Responsive Ecologies Exhibition

Come checkout the Waterman’s Art Centre from the 6th of December until the 21st of January for an immersive and interactive visual experience entitled “Responsive Ecologies” developed in collaboration with artists captincaptin. We will also be giving a talk on the 10th of December from 7 p.m. – 9 p.m. during CINE: 3D Imaging in Art at the Watermans Center.

Responsive Ecologies is part of a wider ongoing collaboration between artists captincaptin, the ZSL London Zoo and Musion Academy. Collectively they have been exploring innovative means of public engagement, to generate an awareness and understanding of nature and the effects of climate change. All of the contained footage has come from filming sessions within the Zoological Society; this coincidentally has raised some interesting questions on the spectacle of captivity, a issue which we have tried to reflect upon in the construction and presentation of this installation. The nature of interaction within Responsive Ecologies means that a visitor to the space cannot simply view the installation but must become a part of its environment. When attempting to perceive the content within the space the visitor reshapes the installation. Everybody has a degree of impact whether directed or incidental, and when interacting as a group it is interesting to see how collective behaviour can develop and incite the outcome of the work.

The installation will encompass participants in 360 degrees with a small door opening. Some pictures of a mock setup using 2 projectors:

And a short video:

Responsive Ecologies Test from pkmital on Vimeo.

Using overhead motion capture and 4 channels of video projection, users are invited to interact in an immersive cinematic environment. The video above shows a demo proof-of-concept for the upcoming installation in Waterman’s Art Centre, London, UK, where artists captincaptin and Parag K Mital have been collaborating to install their piece, “Responsive Ecologies” for their residency the next two months. The work occurs as part of a larger on-going collaboration with ZSL London Zoo and the Musion Academy.

Earlier I posted a video showing how the motion detection works:

Overhead Blob Detection from pkmital on Vimeo.

Using OpenCV with openFrameworks to do blob detection for an upcoming installation entitled “Responsive Ecologies” at the Waterman’s Art Centre in London, UK in collaboration with artists captincaptin.

Full exhibition text follows:

As part of a system of numerous dynamic connections and networks, we are reactive and deterministic to a complex system of cause and effect. The consequence of our actions upon our selves, the society we live in and the broader natural world is conditioned by how we perceive our involvement. The awareness of how we have impacted on a situation is often realised and processed subconsciously, the extent and scope of these actions can be far beyond our knowledge, our consideration, and importantly beyond our sensory perception. With this in mind, how can we associate our actions, many of which may be overlooked as customary, with for instance, the honey bee depopulation syndrome or the declining numbers of Siberian Tigers?

The expanding distance between our lives and the natural world has created a detachment in which we begin to lose the emotion of responsibility. The exhibition references this human-nature dislocation, creating a dynamic environment which can be effeced and controlled by the audience. It aims to symbolise an embodiment of ecology, exploring the relations of organisms, and their interactions with the environment.

Responsive Ecologies is part of an ongoing collaboration with ZSL London Zoo and Musion Academy. Collectively we have been exploring innovative means of public engagement, to generate an awareness and understanding of nature and the effects of climate change. All of the contained footage has come from filming sessions within the Zoological Society; this coincidentally has raised some interesting questions on the spectacle of captivity, a issue which we have tried to reflect upon in the construction and presentation of this installation. The nature of interaction within Responsive Ecologies means that a visitor to the space can not simply view the installation but must become a part of its environment. When attempting to perceive the content within the space the visitor reshapes the installation. Everybody has a degree of impact whether directed or incidental, and when interacting as a group it is interesting to see how collective behaviour can develop and incite the outcome of the work.

Gareth Goodison and Jonathan Munro have been working collaboratively as captincaptin since 2007, creating installations and sculptures, which interact and respond to public presence, to question the role of audience participation in the display and creation of contemporary art. They have presented their work at various new media events including Future Everything Festival, Abandon Normal Devices, Leeds Expo, and the V&A. Parag K Mital is a computational visual artist and PhD student at Goldsmiths Computing Dept. investigating questions of liminality, perception, and attention in mixed reality environments.

The exhibition continues outside the theatre, presenting more outcomes of this collaboration, including the work by artists Madi Boyd and Kira Zhigalina. Special thanks to ZSL London Zoo and Musion Academy.

http://www.watermans.org.uk/exhibitions/exhibitions/future-exhibitions/responsive-ecologies.aspx

6DOF Head Tracking

The following demo works with SeeingMachines FaceAPI in openFrameworks controlling a Mario avatar.  It also has some really poor gesture recognition (and learning but it’s not shown here), though a threshold on the rotation DOF would have produced better results for the simple task of looking up/down left/right gestures.

6DOF Head Tracking from pkmital on Vimeo.

interfacing seeingmachines faceapi with openFrameworks to control a 3D mario avatar

This is just with the non-commercial license. The full commercial license (~$3000?) gives you access to lip/mouth tracking and eye-brows, as well as much more flexibility in how to use their api with different/multiple cameras and accessing image data.

Of course, there are other initiatives at producing similar results. Mutual information based template trackers, for instance, seem to be state-of-art. Take a look at recent work by Panin and Knoll using OpenTL:

 

I imagine a lot of people would like this technology.

Keyframe based modeling

Playing with MSERs in trying to implement an algorithm for feature-based object tracking.  The algorithm first finds MSERs, warps them to circles, describes them with a SIFT descriptor, and then indexes keyframes of sift vectors by using vocabulary trees.   Of course that’s a ridiculously simplified explanation, but look at what it’s capable of!!!:

 

Microsoft Kinect

This is big.  In less than a week, the Kinect has been hacked and ported for windows, osx, linux, java and processing, max/msp (almost), and flash…

Much much more to come: 

“Memory” Video @ AVAF 2010

Please rate, share, and comment!

Memory @ AVAF 2010 from pkmital on Vimeo.

‘Memory’ is an augmented installation of a neural network by Parag K Mital & Agelos Papadakis.
hand blown glass, galvanized metal chain, projection, cameras; 1.5m x 2.5m x 3m

Ghostly images of faces appear as recorded movie clips within neural-shaped hand-blown glass pieces. As one begins to look at the neurons, they notice the faces as their own, trapped as disparate memories of a neural network.

Filmed and installed for the Athens Video Art Festival in May 2010 in Technopolis, Athens, Greece. The venue is a disused gas factory converted art space.

Also seen at Kinetica Art Fair, Ambika P3, London, UK, 2010; Passing Through Exhibition, James Taylor Gallery, London, UK, 2009; Interact, Lauriston Castle, Edinburgh, UK, 2009.

DSC_0466.jpg
DSC_0551.jpg DSC_0420.jpgDSC_0591.jpgDSC_0569.jpg

Facebook Graph API

If you are one of the +500 million users of facebook, and you know your user id, try plugging it in here: http://zesty.ca/facebook/

This uses the Facebook Graph API to get information about Facebook users in a very accessible manner.  Of course, it is only your “public” information that is accessible without authorization.  But once you “allow” an application to access your information, you’re allowing access to EVERYTHING.

Generally, these items are publicly known:

{
   "id": "0123456789",
   "name": "Parag K Mital",
   "first_name": "Parag",
   "middle_name": "K",
   "last_name": "Mital",
   "locale": "en_US"
}

and also your Profile picture.

Check out a montage of the first 3600 Facebook user’s profile pictures, obtained just by using the public url: http://graph.facebook.com/USER_ID/picture


And an image of the average of all 3600 profile images:

X-Ray @ the Roxy Arthouse

Come to the Roxy Arthouse on June 10 for Neverzone and June 26 for Is This a Test? where I’ll be presenting my latest installation X-Ray, as well as to catch other brilliant Scotland-based artists.  More info on the flyers below.


Neverzone / Thursday June 10th 19:00-23:00


Is This a Test? / Saturday June 26th 19:00-23:00

[update] Plug for Is This a Test? on creativeboom.co.uk

[update 2] video now online:

X-RAY from pkmital on Vimeo.




Copyright © 2010 Parag K Mital. All rights reserved. Made with Wordpress. RSS