Learning Human Optical Flow

Abstract

The optical flow of humans is well known to be useful for the analysis of human action. Given this, we devise an optical flow algorithm specifically for human motion and show that it is superior to generic flow methods. Designing a method by hand is impractical, so we develop a new training database of image sequences with ground truth optical flow. For this we use a 3D model of the human body and motion capture data to synthesize realistic flow fields. We then train a convolutional neural network to estimate human flow fields from pairs of images. Since many applications in human motion analysis depend on speed, and we anticipate mobile applications, we base our method on SpyNet with several modifications. We demonstrate that our trained network is more accurate than a wide range of top methods on held-out test data and that it generalizes well to real image sequences. When combined with a person detector/tracker, the approach provides a full solution to the problem of 2D human flow estimation. Both the code and the dataset are available for research.

Downloads

This page provides the datasets for the paper Learning Human Optical Flow (2018) and Learning Multi-Human Optical Flow (2019). The former dataset provides videos of a single synthetic human moving in front of a random background and the respective optical flow. The latter provides a dataset of multiple synthetic humans moving in front of random backgrounds including the optical flow and a fine-grained human part segmentation.

For the multi-human dataset we provide 2D multi-person pose annotation, camera blur parameters, the camera matrix, the depth map, gender tags, normal maps, object Id maps, the SMPL+H pose coefficients, 3D joint locations, an occlusion label for each joint (heuristic), a scale parameter, body part segmentation maps, SMPL+H shapes, global translation for each synthetic human and the z-rotation of each synthetic human. A more detailed description of these ground truth modalities can be found here.

To download these datasets please register on this website. After logging in you will find the links in the download section.

10 Dec 2019:

The dataset currently includes paired RGB images and ground truth optical flow.

17 Jan 2020:

We have released the training and testing code.
The code to generate data yourself can be found here. Furthermore, this Git Repository contains demo code to load and processs the ground truth modalities.

19 March 2020:

We added more ground truth modalities to the download section. The dataset now provides flow, 2D multi-person pose annotation, camera blur parameters, the camera matrix, the depth map, gender tags, normal maps, object Id maps, the SMPL+H pose coefficients, 3D joint locations, an occlusion label for each joint (heuristic), a scale parameter, body part segmentation maps, SMPL+H shapes, global translation rotation for of each synthetic human

Referencing the Dataset

If you use one of the datasets please cite the respective paper.

To cite the Multi-Human Optical Flow dataset please cite:

@article{MultiHumanflow,
		title = {Learning Multi-Human Optical Flow},
		author = {Ranjan, Anurag and Hoffmann, David T. and Tzionas, Dimitrios and Tang, Siyu and Romero, Javier and Black, Michael J.},
		journal = {International Journal of Computer Vision (IJCV)},
		month = jan,
		year = {2020},
		url = {https://humanflow.is.tue.mpg.de},
		month_numeric = {1},
		doi = {10.1007/s11263-019-01279-w},
		}

To cite the Single-Human Optical Flow dataset please cite:

@inproceedings{humanflow,
		title = {Learning Human Optical Flow},
		author = {Ranjan, Anurag and Romero, Javier and Black, Michael J.},
		booktitle = { 29th British Machine Vision Conference},
		month = sep,
		year = {2018},
		url = {https://github.com/anuragranj/humanflow},
		month_numeric = {9}
		}