Generative Adversarial Network
Performs interactive audio synthesis using a previously trained auto encoder. VIsualizes hidden layer and allows for interaction with hidden units. W/ Andy Sarroff. See Andy's github (woodshop) for more interesting projects using Deep Learning and audio synthesis.
Audio and Music Analysis and Synthesis in Python
Course materials/Homework materials for the FREE MOOC course on "Creative Applications of Deep Learning w/ Tensorflow" #CADL
Caffe models in TensorFlow
CARPE is "Computational Algorithmic Representation and Processing of Eye-movements". It supports visualizations of film/eye-movements in columnar format, peak-through visualizations, heatmaps, optical flow analysis, gaze clustering, and many other visualization options. See http://thediemproject.wordpress.com for more details. Compuational and Algorithmic Representation and Processing of Eye-movements - OSX Version
CARPE is "Computational Algorithmic Representation and Processing of Eye-movements". It supports visualizations of film/eye-movements in columnar format, peak-through visualizations, heatmaps, optical flow analysis, gaze clustering, and many other visualization options. See http://thediemproject.wordpress.com for more details. Initial release of CARPE hosted for historical purposes. Includes many experimental addons for CARPE including dROIs, GMM, GPU-Flow, and others. See also CARPE and NSCARPE for more recent, stable versions not including the experimental addons.
openFrameworks projects from my course at Srishti's School of Art, Design and Technology's Center for Experimental Media: http://pkmital.com/home/teaching/cema-workshop/
(unfinished) storing data in leveldb
Matlab scripts specific to the DIEM database
testing eeg encoding/decoding. this produces some gabor wavelets and co-registers the visualization and recording of eeg.
For streaming/logging emotiv eeg headset data
testing some encoding/decoding using EEG. this produces various animations that are co-registered with the recording of eeg.
Uses cURL to download from Freesound.org
Does finger tracking, contour analysis, and different types of shape description for the hand contour using a 3D ROI around the hand-tracked by NITE middleware, using OpenCV 2.2, an OpenGL scene, and OpenNI.
LuminOrder: An OS X app to reorder a video in order of its frames' brightness levels.
Multiscale Visualization ToolKit
Materials for "Audiovisual Interaction w/ Machine Learning", Bachelors (416) and Graduate (616) course @ CalArts MTIID MTEC
openFrameworks addon for visualizing and interfacing with pre-trained models in Caffe: Convolutional Architectures for Fast Feature Embedding. Requires Caffe, openFrameworks 64-bit, glog, hdf5, OpenCV, CUDA, pkmMatrix, and pkmHeatmap. Pre-trained models not included but can be found linked in Caffe's "Model Zoo" and placed in the bin/data directory of the example project.
simple way to stream 32-bit float data from itunes in real-time
ofxOpenCV linking against OpenCV 188.8.131.52, including libraries for OSX
Ph.D. Thesis from Goldsmiths, University of London entitled, "Audiovisual Scene Synthesis". Hosts all images and latex files.
Sound synthesis library. Implements a number of feature databses including frame-based, segment-based, and sequence-based audio analysis/storing/retrieval. Also includes libraries for GPS-based synthesis including binarualization retrieval. Complements many other libraries including pkmBinaural and Memory Mosaic. Heavy dependence on vectorized ops using pkmMatrix and OSX's library Accelerate.
Background modeling for foreground subtraction, tracks multiple blobs (people), their orientations (using leading motion vector), and has a nice visual display for seeing the results... video demonstration here: http://vimeo.com/22054133 - more info here: http://pkmital.com
track overhead using color and map tracked points to a new geometry using a homography transformation and calibration routine - some example test videos are provided in the bin/data directory of an overhead capture. The tracking transformation is useful for when you need a defined metric space of your tracking parameters, or need to account for different user heights in tracking their paths in a space.
3d Object Tracking and Pose Estimation for the iPhone
Interfacing libcluster for doing Variational Dirichlet Process Gaussian Mixture Models. Depends on Eigen3 and pkmMatrix. Libcluster included.
More projects are listed on my github