I’ve been working on developing a method for automatic head-pose tracking, and along the way have come to model facial appearances. I start by initializing a facial bounding box using the Viola-Jones detector, a well known and robust detector used for training objects. This allows me to centralize the face. Once I know where the 2D plane of the face is in an image, I can register an Active Shape Model like so:
After multiple views of the possible appearance variations of my face, including slight rotations, I construct an appearance model.
The idea I am working with is using the first components of variations of this appearance model for determining pose. Here I show the first two basis vectors and the images they reconstruct:
As you may notice, these two basis vectors very neatly encode rotation. By looking at the eigenvalues of the model, you can also interpret pose.
Archived entries for eye movements
Facial Appearance Modeling/Tracking
6DOF Head Tracking
The following demo works with SeeingMachines FaceAPI in openFrameworks controlling a Mario avatar. It also has some really poor gesture recognition (and learning but it’s not shown here), though a threshold on the rotation DOF would have produced better results for the simple task of looking up/down left/right gestures.
6DOF Head Tracking from pkmital on Vimeo.
interfacing seeingmachines faceapi with openFrameworks to control a 3D mario avatar
This is just with the non-commercial license. The full commercial license (~$3000?) gives you access to lip/mouth tracking and eye-brows, as well as much more flexibility in how to use their api with different/multiple cameras and accessing image data.
Of course, there are other initiatives at producing similar results. Mutual information based template trackers, for instance, seem to be state-of-art. Take a look at recent work by Panin and Knoll using OpenTL:
I imagine a lot of people would like this technology.
DIEM Website
The DIEM Project (Dynamic Images and Eye Movements) has a sleek new website which you can check out here: http://www.psy.ed.ac.uk/diem
CLOSE-UP 2
An event organized by the new Center for Film, Performance, and Media Arts (CFPMA) was held today in Edinburgh University discussing recent topics in… film, performance, and media arts.
It was an interesting group of people that strangely somehow all had much in common. I had the fortune of presenting my research as it relates to DIEM in the place of Tim J. Smith.
The schedule:
Close-Up 2: Schedule for Wednesday 17th June 2009
10am coffee and tech checks (G.11, William Robertson Building)
10.30 Welcome, Annette Davison (Music, ACE and Director, Cfpma), Martine Beugnet (LLC, Convener of Film Studies)
Who is who, where is what? People and resources for which the Cfpma will provide a point of convergence.
11am Individual Presentations (MAX 10 mins each):
Andrew Lawrence (African Studies, SSPS) — Difficult satire under austerity: the films of Sissako and Amoussou
Martine Beugnet (Film, LLC) — “The Wounded Screen”
Richard Williams (Architecture, ACE) — “The Modernist City on Film”
Stephen Cairns (Architecture, ACE) — “Cultures of Legibility: Emergent Urban Landscapes in Southeast Asia”
Simon Frith and Annette Davison (Music, ACE) — “The Role of Cinemas in the History of Live Music”
Mary Fogarty (Music, ACE) — “The New Silent Cinema in Live”
Yoko Matsumoto-Sturt (Asian Studies, LLC) — “Japanese language and culture in J-pop”
Kriss Ravetto (Film, LLC) — “Spectres of the Digital”
Eric Laurier (Geosciences) — “Agreements in Editing”
Parag Mital (Visual Cognition Lab, PPLS) — “Dynamic Images and Eye Movements”
Smita Kheria (Law) — “Copyright law and new media art
Lunch: 1-2pm
2pm – joint presentations
Richard Coyne, Penny Travlou, Mark Wright (ACE, ECA, and Informatics) — “Emerging forms of digital media and the democratization of urban discourse”
Jolyon Mitchell, Alina Birzache, Milja Radovic, Yasmin Fedda (Divinity) — “Seeing Through Film, Religion and Ethics
3pm COFFEE
3.30pm DISCUSSION FORUM 1
Cross-Subject/School /Institution teaching and research supervision: current/future plans
Chair: Kriss Ravetto
With Martine Beugnet, Sarah Colvin, John Lee, Fiona Littleton, Martine Pierquin
4pm INFO and UPDATES
Knowledge Transfer and Exchange: Anne-Sofie Laegran
Cfpma Web Presence: Annette Davison
‘Film in the Public Space’ and the Roberts Fund: Martine Beugnet
4.20pm DISCUSSION FORUM 2
The future of the Cfpma.
Chair: Annette and Martine
What could the Centre usefully do/encourage to support its members and foster research collaboration?
5pm BREAK – and move to David Hume Tower conference room
5.15pm Professor Tim Lenoir, CONTEMPLATING SINGULARITY
Cinet (Cinema Network) talk, followed by wine reception and launch of CFPMA
The talk explores how the postbiological and posthuman future has haunted cultural studies of technoscience for two decades. Concern (and in some quarters enthusiasm) that contemporary technoscience is on a path leading beyond simple human biological improvements and prosthetic enhancements to a complete human makeover has been sustained by the exponential growth in power and capability of computer technology since the early 1990s. The deeper fear is that somehow digital code and computer-mediated communications are getting under our skin, and in the process we are being transformed.
Tim Lenoir is the Kimberly Jenkins Chair for New Technologies and Society at Duke University. He has published several books and articles on the history of biomedical science from the nineteenth century to the present. His more recent work has focused on the introduction of computers into biomedical research from the early 1960s to the present, particularly the development of computer graphics, medical visualization technology, the development of virtual reality and its applications in surgery and other fields. Lenoir has also been engaged in constructing online digital libraries for a number of projects, including an archive on the history of Silicon Valley. Two recent projects include a web documentary project to document the history of bioinformatics funded by the Bern Dibner and Alfred P. Sloan Foundations, and How They Got Game, a history of interactive simulation and video games. With economists Nathan Rosenberg, Henry Rowen, and Brent Goldfarb he has just completed a collaborative study for Stanford University on Stanford’s historical relationship to Silicon Valley entitled, Inventing the Entrepreneurial Region: Stanford and the Co-Evolution of Silicon Valley. In support of these projects, Lenoir has developed software tools for interactive web-based collaboration. In this connection he is currently engaged with colleagues at UC Santa Barbara in developing the NSF-supported Center for Nanotechnology in Society, where he contributes to the effort to document the history, societal, and ethical implications of bionanotechnology.
There were those interested in the slides of my presentation and I thought to include it online:
The videos shown in the slides are available online on our Vimeo and Youtube accounts:
Eye movments during the Video Republic from VisCogEdinburgh on Vimeo.
Eye movements during the Wimbeldon Macenroe-Bjorn Borg match from VisCogEdinburgh on Vimeo.
Eye Movements during 50 People 1 Question (Brooklyn) from VisCogEdinburgh on Vimeo.
See many more here and here (these are constantly being updated with more videos).
Eye movements during video
Part of my research entails investigating ways of visualizing eye-movement data during dynamic images.
Some vids:
And one of President Obama’s inaugural speech:
from VisCogEdinburgh