Emotion recognition based on physiological changes in music listening

• Published in: IEEE transactions on pattern analysis and machine intelligence Vol. 30; no. 12; pp. 2067 - 2083
• Main Authors: Jonghwa Kim, Andre, Elisabeth
• Format: Journal Article
• Language: English
• Published: Los Alamitos, CA IEEE 01.12.2008; IEEE Computer Society; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Adaptation, Physiological - physiology; Algorithms; and processing; Applied sciences; Arousal; Arousal - physiology; Artificial Intelligence; Auditory Perception - physiology; Biosensors; Channels; Classification; Classifier design and evaluation; Computer science; control theory; systems; Computer systems and distributed systems. User interface; Conductivity measurement; Discriminant analysis; Disk recording; Emotion recognition; Emotions; Emotions - physiology; Entropy; Exact sciences and technology; Feature evaluation and selection; Frequency; Human-centered computing; Humans; Intelligence; Interaction styles; Laboratories; Methodologies and techniques; Monitoring, Physiologic - methods; Music; Pattern analysis; Pattern Recognition, Automated - methods; Pattern recognition. Digital image processing. Computational geometry; Recognition; Robotics; Signal analysis; Signal processing; Skin; Software; Speech; Studies; synthesis; Theory and methods; User/Machine Systems; synthesis; and processing; Human-centered computing; Feature evaluation and selection; User/Machine Systems; Theory and methods; Classifier design and evaluation; Robotics; Signal processing; Methodologies and techniques; Interaction styles; Signal analysis; Pattern analysis; Autonomous system; Time frequency domain method; Man machine system; affective computing; respiration; Modeling; electromyogram; Audiovisual; Induction method; electrocardiogram; User interface; Musical acoustics; Classification; Facies; Electrocardiography; Audiovisual equipment; Time analysis; Pattern extraction; physiological signal; Emotion recognition; Discriminant analysis; autonomic nervous system; skin conductance; biosignal; Frequency domain method; Emotion emotionality; valence; human-computer interaction; Multiscale method; Biosensor; Music; musical emotion; Skin; Automatic recognition; Artificial intelligence; Multiclass; Facial expression; arousal
• ISSN: 0162-8828; 2160-9292; 1939-3539
• DOI: 10.1109/TPAMI.2008.26

Little attention has been paid so far to physiological signals for emotion recognition compared to audiovisual emotion channels such as facial expression or speech. This paper investigates the potential of physiological signals as reliable channels for emotion recognition. All essential stages of an automatic recognition system are discussed, from the recording of a physiological data set to a feature-based multiclass classification. In order to collect a physiological data set from multiple subjects over many weeks, we used a musical induction method that spontaneously leads subjects to real emotional states, without any deliberate laboratory setting. Four-channel biosensors were used to measure electromyogram, electrocardiogram, skin conductivity, and respiration changes. A wide range of physiological features from various analysis domains, including time/frequency, entropy, geometric analysis, subband spectra, multiscale entropy, etc., is proposed in order to find the best emotion-relevant features and to correlate them with emotional states. The best features extracted are specified in detail and their effectiveness is proven by classification results. Classification of four musical emotions (positive/high arousal, negative/high arousal, negative/low arousal, and positive/low arousal) is performed by using an extended linear discriminant analysis (pLDA). Furthermore, by exploiting a dichotomic property of the 2D emotion model, we develop a novel scheme of emotion-specific multilevel dichotomous classification (EMDC) and compare its performance with direct multiclass classification using the pLDA. An improved recognition accuracy of 95 percent and 70 percent for subject-dependent and subject-independent classification, respectively, is achieved by using the EMDC scheme.