Learning Multi-human Optical Flow

• Published in: International journal of computer vision Vol. 128; no. 4; pp. 873 - 890
• Main Authors: Ranjan, Anurag, Hoffmann, David T., Tzionas, Dimitrios, Tang, Siyu, Romero, Javier, Black, Michael J.
• Format: Journal Article
• Language: English
• Published: New York Springer US 01.04.2020; Springer; Springer Nature B.V
• Subjects: Analysis; Artificial Intelligence; Computer Imaging; Computer Science; Datasets; Human motion; Image Processing and Computer Vision; Information management; Motion capture; Optical communication; Optical flow (image analysis); Pattern Recognition; Pattern Recognition and Graphics; Pedestrians; Special Issue on Machine Vision with Deep Learning; Three dimensional models; Training; Vision; Learning; Human; Synthetic; Optical; Dataset; Bodies; Deep; Flow; Humanflow
• ISSN: 0920-5691; 1573-1405
• DOI: 10.1007/s11263-019-01279-w

The optical flow of humans is well known to be useful for the analysis of human action. Recent optical flow methods focus on training deep networks to approach the problem. However, the training data used by them does not cover the domain of human motion. Therefore, we develop a dataset of multi-human optical flow and train optical flow networks on this dataset. We use a 3D model of the human body and motion capture data to synthesize realistic flow fields in both single- and multi-person images. We then train optical flow networks to estimate human flow fields from pairs of images. We demonstrate that our trained networks are more accurate than a wide range of top methods on held-out test data and that they can generalize well to real image sequences. The code, trained models and the dataset are available for research.