Distracted driving detection based on the fusion of deep learning and causal reasoning

• Published in: Information fusion Vol. 89; pp. 121 - 142
• Main Authors: Ping, Peng, Huang, Cong, Ding, Weiping, Liu, Yongkang, Chiyomi, Miyajima, Kazuya, Takeda
• Format: Journal Article
• Language: English
• Published: Elsevier B.V 01.01.2023
• Subjects: Behavior recognition; Causal reasoning; Counterfactual reasoning; Deep learning network; Driving distraction detection; Counterfactual reasoning; Causal reasoning; Behavior recognition; Driving distraction detection; Deep learning network
• ISSN: 1566-2535; 1872-6305
• DOI: 10.1016/j.inffus.2022.08.009

Distracted driving is one of the key factors that cause drivers to ignore potential road hazards and then lead to accidents. Existing efforts in distracted behavior recognition are mainly based on deep learning (DL) methods, which identifies distracted behaviors by analyzing static characteristics of images. However, the convolutional neural network (CNN) — based DL methods lack the causal reasoning ability for behavior patterns. The uncertainty of driving behaviors, noise of the collected data, and occlusion between body agents, bring additional challenges to existing DL methods to recognize distracted behaviors continuously and accurately. Therefore, in this paper, we propose a distracted behavior recognition method based on the Temporal–Spatial double-line DL network (TSD-DLN) and causal And-or graph (C-AOG). TSD-DLN fuses the attention feature extracted from the dynamic optical flow information and the spatial feature of the single video frame to recognize the distracted driving posture. Furthermore, a causal knowledge fence based on C-AOG is fused with TSD-DLN to improve the recognition robustness. The C-AOG represents the causality of behavior state fluent change and adopts counterfactual reasoning to suppress behavior recognition failures caused by frame features distortion or occlusion between body agents. We compared the performance of the proposed method with other state-of-the-art (SOTA) DL methods on two public datasets and self-collected dataset. Experimental results demonstrate that proposed method significantly outperforms other SOTA methods when acquiring distracted driving behavior by processing consecutive frames. In addition, the proposed method exhibits accurate continuous recognition and robustness under incomplete observation scenarios. •A novel behavior recognition model based on multi-features fusion is proposed.•A hierarchical causality graph is designed for behavior fluent inferring.•Counterfactual reasoning mechanism for misrecognition suppression is proposed.•A new behavior dataset representing the distracted driving process is constructed.