Parag Kumar Mital

In 2007, a very nice implementation of a variational implementation of optical flow was described in: A Duality Based Approach for Realtime TV-L1 Optical Flow by C. Zach et. al. I won’t get into the details too much, but the formulation is described by this equation:

E = int_?{?|I0(x) ? I1(x + u(x))| + |?u| dx}

If you are familiar with the seminal work of Horn and Schunck, you will notice it is fairly similar to their variational formulation:

min_u{ int_?{ (|?u1|^2 + |?u2|^2) d? } + ? int_?{ ((I1(x + u(x)) ? I0(x))^2) d? }

And although it looks incredibly simple now, it is in fact fairly difficult computationally since now both terms are not continuously differentiable. To overcome this difficulty, they follow the work of Rudin-Osher-Fatemi energy for total variation image denoising.

Another big contribution comes in their implementation on the GPU. By linearization of the generally non-convex energy functional shown above, the problem is reduced to a pixel-wise convex energy minimization problem. Additionally, by employing coarse-to-fine image pyramids, they are able to account for both small and large movements. Luckily, graphics cards are great at doing both of these sorts of computations very quickly. You can find a release of their implementation here.

In 2009 at the upcoming British Machine Vision Conference, Manuel Werlberger et. al will detail a new formulation which introduces an anisotropic regularization based on the robust Huber norm instead of the total variation regularization term. This extension derives from a key observation: motion discontinuities often occur along object boundaries. By using image-derived information from large gradients, discontinuities from over-smoothing along object boundaries are preserved.

Manuel has also released his code on the aforementioned website. However, since I have been stuck in the world of Matlab, I decided to work on an interface to this library encased in a Matlab mex file. You will need to have installed CUDA, Matlab 2008b or higher, and have successfully installed the CUDA plugin.

For now, it suffers one very big problem: you will receive a segmentation fault if you remove the function. Though, I have been in contact with Manuel and hopefully we’ll figure it out. I’ll post back here on any developments.

[UPDATE] No contact from Manuel in awhile but I suspect he must be very busy. I have played with this library enough now to see that there a a few memory problems within the flowlib library itself not appropriately deleting device memory. There is still a problem with the flowlib destructor on win32 as well. If you are going to calculate optical flow on many images, you will most likely run into the same problems I did. Otherwise, you should not have any problems.

[UPDATE 2] The newest source on Manuel’s website is rock solid!  Get it!