Fallen person detection for autonomous driving

• Published in: Expert systems with applications Vol. 213; p. 119242
• Main Authors: Lee, Suhyeon, Lee, Sangyong, Seong, Hongje, Hyun, Junhyuk, Kim, Euntai
• Format: Journal Article
• Language: English
• Published: Elsevier Ltd 01.03.2023
• Subjects: Autonomous vehicles; Corner cases; Data synthesis; Detection; Fallen person; Corner cases; Detection; Data synthesis; Autonomous vehicles; Fallen person
• ISSN: 0957-4174; 1873-6793
• DOI: 10.1016/j.eswa.2022.119242

Fallen person detection (FPD) is a new problem that aims to detect a person who lies or falls down on driving roads. The biggest difficulty in FPD is capturing a sufficient number of training images of people lying on driving roads because of the dangers involved. In this paper, we propose a novel fallen person detection image synthesis framework to address this difficulty. Our framework first embeds a fallen person instance into an image of a driving road, thereby generating a hard-to-acquire image (image of a person who has fallen on a road) from two easy-to-acquire images (driving road image and fallen person image). We then reduce the domain gap between the two images using domain adaptation. Finally, we remove some pixel artifacts from the border between the fallen person and background area in the synthesized image. Our proposed framework addresses the lack of training data, which is a serious problem inherent to FPD. Furthermore, we develop a new dataset named FPD (Fallen Person detection with Driving scenes)-set to train a detection network. FPD-set consists of four subsets: (1) RealFP218, (2) RealD1.8K, (3) RealFPDK1.4K and (4) RealFPDY1.1K. RealFP218 consists of 218 images of real fallen persons and their pixel-level mask annotations; and RealD1.8K consists of 1820 real driving road images. The two sets will be used to synthesize the driving road images including fallen persons. RealFPDK1.4K and RealFPDY1.1K are test sets which are captured at two different places (K-City and Yonsei University). The two test sets consist of 1400 and 1161 images of real fallen persons on the road with bounding box annotations, respectively. Our dataset covers a variety of conditions, including occlusion, lack of lighting, and shadows, thereby facilitating qualitative and quantitative evaluations in the real world. We released this dataset for the benefit of the autonomous driving society. We verify the effectiveness of our training image synthesis method by applying the detector to the RealFPDK1.4K and RealFPDY1.1K datasets. Our approach achieves AP scores of 0.815 and 0.753, and the scores are higher than those of the baseline by +0.287 and +0.210 on RealFPDK1.4K and RealFPDY1.1K, respectively. Experimental results demonstrate that our framework contributes significantly to training an FPD network. •This paper tackled a new problem, Fallen Person Detection (FPD).•FPD aims to detect a person who lies or falls down on driving roads.•This is the first study on FPD on real driving roads for application to AV.•The image synthesis framework that generates training data is proposed to realize FPD.•A new dataset called FPD-set is constructed and released online.