RFDANet: an FMCW and TOF radar fusion approach for driver activity recognition using multi-level attention based CNN and LSTM network

• Published in: Complex & intelligent systems Vol. 10; no. 1; pp. 1517 - 1530
• Main Authors: Gu, Minming, Chen, Kaiyu, Chen, Zhixiang
• Format: Journal Article
• Language: English
• Published: Cham Springer International Publishing 01.02.2024; Springer Nature B.V; Springer
• Subjects: Accident prevention; Activity recognition; Behavior; Branch-attention-CNN-LSTM network; Complexity; Computational Intelligence; Computer vision; Data Structures and Information Theory; Deep learning; Driver activity recognition; Driver behavior; Engineering; Feature extraction; FMCW radar; Machine learning; Original Article; Radar; Roads; Sensor fusion; Teaching methods; TOF radar; Traffic accidents; Traffic safety; Sensor fusion; Driver activity recognition; TOF radar; Branch-attention-CNN-LSTM network; FMCW radar
• ISSN: 2199-4536; 2198-6053
• DOI: 10.1007/s40747-023-01236-8

Dangerous driving behavior is a major contributing factor to road traffic accidents. Identifying and intervening in drivers’ unsafe driving behaviors is thus crucial for preventing accidents and ensuring road safety. However, many of the existing methods for monitoring drivers’ behaviors rely on computer vision technology, which has the potential to invade privacy. This paper proposes a radar-based deep learning method to analyze driver behavior. The method utilizes FMCW radar along with TOF radar to identify five types of driving behavior: normal driving, head up, head twisting, picking up the phone, and dancing to music. The proposed model, called RFDANet, includes two parallel forward propagation channels that are relatively independent of each other. The range-Doppler information from the FMCW radar and the position information from the TOF radar are used as inputs. After feature extraction by CNN, an attention mechanism is introduced into the deep architecture of the branch layer to adjust the weight of different branches. To further recognize driving behavior, LSTM is used. The effectiveness of the proposed method is verified by actual driving data. The results indicate that the average accuracy of each of the five types of driving behavior is 94.5%, which shows the advantage of using the proposed deep learning method. Overall, the experimental results confirm that the proposed method is highly effective for detecting drivers’ behavior.