Learning-based estimation of operators’ psycho-physiological state

• Published in: Expert systems with applications Vol. 276; p. 127097
• Main Authors: Piccinin, Lisa, Leoni, Jessica, Villa, Eugenia, Milani, Sabrina, Breschi, Valentina, Tanelli, Mara, Colavincenzo, Manuel, Martorana, Stefano
• Format: Journal Article
• Language: English
• Published: Elsevier Ltd 01.06.2025
• Subjects: Multivariate approach; Psycho-physiological state; Safety-oriented system; Scorecard; Stress detection; Unsupervised machine learning; Psycho-physiological state; Unsupervised machine learning; Stress detection; Safety-oriented system; Scorecard; Multivariate approach
• ISSN: 0957-4174
• DOI: 10.1016/j.eswa.2025.127097

Operating complex vehicles, such as cars or aircraft, demands constant attention and can significantly impact the operator’s condition and decision-making. However, accurately assessing the operator state in real-time presents significant challenges due to the complex interplay of physiological, behavioral, and telemetry data. Existing methods often rely on limited data sources, supervised learning approaches (that are sensitive to biased and limited ground truth labels), and small, homogeneous participant sets, hindering performance and generalization capabilities. To overcome these limitations, this study proposes a novel unsupervised machine learning framework for real-time operator state assessment. By integrating diverse data sources, including physiological, behavioral, and telemetry information, and employing an unsupervised approach, the framework can identify stress-related patterns within the data considering the whole picture of the system and without relying on subjective labels. Furthermore, the framework leverages data from a diverse population of operators to account for intra-subject variability. To evaluate its effectiveness, we compared model-derived estimates with operator perceptions using dedicated questionnaires across two distinct simulated scenarios: driving and flight. The results demonstrate the framework’s ability to accurately capture operator state, aligning well with subjective assessments and exhibiting strong generalization across different operating conditions. This research represents a significant step towards the development of robust and reliable real-time operator state monitoring systems. •A multimodal approach to enhance robustness compared to the state-of-the-art.•A ML algorithm to automatically monitor the user’s psycho-physiological state.•Ad hoc feature selection to improve the effectiveness of the classifier.•Custom-designed scorecards to evaluate user performance throughout the entire trial.•An experimental setup with multiple volunteers to evaluate the proposed approach.