Projects

Hand Pose Estimation with Depth Camera

Hand pose estimation is an important research topic in human-computer interaction which has various applications, such as gesture recognition and animation synthesis. Previously the specialized hardware, e.g. the optical sensors and the data-gloves, are commonly used to accomplish this task. Although they provide accurate measurements and achieve real-time performance, such devices are cumbersome to use and expensive. Thus the vision-based methods have been the mainstream in this field, which are cheaper and provide more natural interaction experiences. However, due to the high flexibility and self-occlusion of the hand, it remains a challenging task to capture the articulated hand motions from the visual inputs. In this project we present various techniques to handle these challenges in hand pose estimation.

Publications

Hui Liang, Junsong Yuan and Daniel Thalmann, Resolving Ambiguous Hand Pose Predictions by Exploiting Part Correlations, in IEEE Trans. Circuits and Systems for Video Technology, 2014. [Video Link]

Hui Liang, Junsong Yuan and Daniel Thalmann, Improved Hand Pose Estimation via Multimodal Prediction Fusion, in Computer Graphics International (CGI), 2014.

Hui Liang, Junsong Yuan, Daniel Thalmann and Zhengyou Zhang, Model-based Hand Pose Estimation via Spatial-temporal Hand Parsing and 3D Fingertip Localization, in the Visual Computer Journal, June 2013. [Video Link]

Hui Liang, Junsong Yuan and Daniel Thalmann, 3D Fingertip and Palm Tracking in Depth Image Sequences, in ACM Int'l Conf. on Multimedia 2012.

NTU Hand Posture Dataset [Link]

The dataset contains totally 1354 real depth images of the right hands of four different subjects captured using a SoftKinetic DS325 camera. The resolution of these images is 320*240. The ground truth 3D positions of the sixteen hand joints in the real dataset are obtained by manual annotation. The detailed specifications and the C++ program to read/view the frames are provided inside the dataset.

Hand Gesture Recognition

Vision-based hand gesture recognition has various applications in human computer interaction such as virtual reality and sign language recognition. Such systems greatly improve the users’ interaction experience by providing a natural and convenient way for interaction between human beings and computers. Despite the achievement in this field, most existing methods are still sensitive to inaccurate hand segmentation results and hand viewpoint variations. In this project, we try to develop a real-time hand gesture recognition that does not require the user to wear any markers and at the same time works accurately and robustly against imperfect inputs.

Publications

Hui Liang, Junsong Yuan and Daniel Thalmann, Parsing the Hand in Depth Images, in IEEE Trans. Multimedia, vol. 16, no. 5, Aug. 2014. [Video Link]

Hui Liang and Junsong Yuan, Hand Parsing and Gesture Recognition with a Commodity Depth Camera, in Computer Vision and Machine Learning with RGB-D Sensors, Springer, 2014.

Projects and Codes [Link]

We present the project to recognize the ten digit hand gestures with a commercial depth camera. This implementation is based on part of our work in the two papers [1, 2], which first uses the method in [1] to parse the hand into 12 different hand parts and extract the joints from the parsed hand parts, and then use the template matching method in [2] to recognize the hand gesture. The project is written in C++/OpenCV and runs in real-time. [Video Link]

Datasets

My friend Mr. Zhou Ren's work on hand gesture recognition in our group can be found here [Link], which is published in the following papers. The dataset used in the paper can be found here [Link], which consists of 10-gesture hand gestures captured with Kinect sensor. It contains both color images and depth maps, and is collected under cluttered background, which includes 10 subjects × 10 gestures/subject × 10 cases/gesture = 1000 cases.

Zhou Ren, Junsong Yuan, Jingjing Meng and Zhengyou Zhang, Robust Part-based Hand Gesture Recognition using Kinect Sensor, in IEEE Trans. on Multimedia, Aug. 2013.

Zhou Ren, Junsong Yuan and Zhengyou Zhang, Robust Hand Gesture Recognition based on Finger-Earth Mover’s Distance with a Commodity Depth Camera, in ACM MM, 2011.

The code for some above projects can also be found in my Github account: https://github.com/shrekei