Towards affect-aware vehicles for increasing safety and comfort: recognising driver emotions from audio recordings in a realistic driving study

• Published in: IET intelligent transport systems Vol. 14; no. 10; pp. 1265 - 1277
• Main Authors: Requardt, Alicia F, Ihme, Klas, Wilbrink, Marc, Wendemuth, Andreas
• Format: Journal Article
• Language: English
• Published: The Institution of Engineering and Technology 01.10.2020
• Subjects: 4‐emotion‐class detection problem; adaptive automotive safety applications; affect‐aware vehicles; aggressive driving behaviours; automatic analysis; chance level; driver emotion recognition; driver information systems; driver in‐time monitoring; emotion recognition; emotional driver states; fatal road accidents; feature extraction; feature selection; feature sets; hyper‐parameter optimisation; in‐car assistance; learning (artificial intelligence); low‐expressed emotions; machine learning algorithms; optimal feature set; optimisation; random forests; realistic driving study; real‐world situations; Research Article; road accidents; road safety; small random forests; speech data; speech feature selection; standard emobase feature set; subject data identification; vehicle safety; emotional driver states; random forests; vehicle safety; road safety; emotion recognition; audio recordings; aggressive driving behaviours; speech data; in-car assistance; optimisation; road accidents; standard emobase feature set; feature extraction; driver information systems; automatic analysis; learning (artificial intelligence); real-world situations; machine learning algorithms; 4-emotion-class detection problem; feature selection; subject data identification; low-expressed emotions; feature sets; speech feature selection; small random forests; chance level; fatal road accidents; affect-aware vehicles; driver in-time monitoring; hyper-parameter optimisation; adaptive automotive safety applications; driver emotion recognition; optimal feature set; audio recording; realistic driving study
• DOI: 10.1049/iet-its.2019.0732
• ISSN: 1751-956X

For vehicle safety, the in-time monitoring of the driver and assessing his/her state is a demanding issue. Frustration can lead to aggressive driving behaviours, which play a decisive role in up to one-third of fatal road accidents. Consequently, the authors present the automatic analysis of the emotional driver states of frustration, anxiety, positive and neutral. Based on experiments with normal drivers within cars in real-world (low expressivity) situations, they use speech data, as speech can be recorded with zero invasiveness and comes naturally in driving situations. A careful selection of speech features, subject data identification, hyper-parameter optimisation, and machine learning algorithms was applied for this difficult 4-emotion-class detection problem, where the literature hardly reports results above chance level. In-car assistance demands real-time computing. A very detailed analysis yields best results with relatively small random forests, and with an optimal feature set containing only 65 features (6.51% of the standard emobase feature set) which outperformed all other feature sets, producing 35.38% unweighted average recall (53.26% precision) with low computational effort, and also reducing the inevitably high confusion of ‘neutral’ with low-expressed emotions. This result is comparable to and even outperforming other reported studies of emotion recognition in the wild. Their work, therefore, triggers adaptive automotive safety applications.