Imitation Learning-Based Algorithm for Drone Cinematography System

• Published in: IEEE transactions on cognitive and developmental systems Vol. 14; no. 2; pp. 403 - 413
• Main Authors: Dang, Yuanjie, Huang, Chong, Chen, Peng, Liang, Ronghua, Yang, Xin, Cheng, Kwang-Ting
• Format: Journal Article
• Language: English
• Published: Piscataway IEEE 01.06.2022; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Algorithms; Animation; Cameras; Cinematography; Cinematography system; Drones; Encoding; Human motion; imitation filming; Machine learning; Servocontrol; Skeleton; Two dimensional displays; unmanned aerial vehicles; Videography; viewpoint control; Virtual environments; Visualization
• ISSN: 2379-8920; 2379-8939
• DOI: 10.1109/TCDS.2020.3043441

Viewpoint selection for capturing human motion is an important task in autonomous aerial videography, animation, and virtual 3-D environments. Existing methods rely on heuristics for selecting the "best" viewpoint, which requires human effort to summarize and integrate viewpoint selection rules into a visual servo system to control a camera. In this work, we propose an integrated aerial filming system for autonomously capturing cinematic shots of action scenes on the basis of a set of demonstrations given for imitation. Our model, which is built on the basis of the deep deterministic policy gradient, takes a sequence of a subject's skeleton and the camera pose as input and outputs the camera motion with an optimal viewpoint related to the subject. In addition, we design a spatial attention network to selectively focus on the discriminative joints of the skeleton within each frame. Given the demonstrations with human motions, our framework learns to predict the next best viewpoint by imitating the demonstrations for viewing the motion of the subject. Extensive experimental results in simulated and real outdoor environments demonstrate that our method can successfully mimic the viewpoint selection strategy and capture a more accurate viewpoint than state-of-the-art autonomous cinematography methods.