Expression of affect in spontaneous speech: Acoustic correlates and automatic detection of irritation and resignation

• Published in: Computer speech & language Vol. 25; no. 1; pp. 84 - 104
• Main Authors: Laukka, Petri, Neiberg, Daniel, Forsell, Mimmi, Karlsson, Inger, Elenius, Kjell
• Format: Journal Article
• Language: English
• Published: Elsevier Ltd 2011
• Subjects: Acoustic features; Acoustic measurement; Acoustics; Automatic speech classification; Classification; Corpus Linguistics; Correlation; Emotion recognition; Emotions; Formants; Fundamental Frequency; Human Computer Communication; Human-computer interaction; HUMANIORA och RELIGIONSVETENSKAP; HUMANITIES and RELIGION; Information technology; Informationsteknik; Intensity; Irritation; Languages and linguistics; Psychology; Psykologi; SAMHÄLLSVETENSKAP; SOCIAL SCIENCES; Socialvetenskap; Speech; Spontaneous Speech; Språkvetenskap; Stimuli; TECHNOLOGY; TEKNIKVETENSKAP; Voice; Spontaneous speech; Emotion recognition; Automatic speech classification; Acoustic features; Human–computer interaction
• ISSN: 0885-2308; 1095-8363; 1095-8363
• DOI: 10.1016/j.csl.2010.03.004

The majority of previous studies on vocal expression have been conducted on posed expressions. In contrast, we utilized a large corpus of authentic affective speech recorded from real-life voice controlled telephone services. Listeners rated a selection of 200 utterances from this corpus with regard to level of perceived irritation, resignation, neutrality, and emotion intensity. The selected utterances came from 64 different speakers who each provided both neutral and affective stimuli. All utterances were further automatically analyzed regarding a comprehensive set of acoustic measures related to F0, intensity, formants, voice source, and temporal characteristics of speech. Results first showed that several significant acoustic differences were found between utterances classified as neutral and utterances classified as irritated or resigned using a within-persons design. Second, listeners’ ratings on each scale were associated with several acoustic measures. In general the acoustic correlates of irritation, resignation, and emotion intensity were similar to previous findings obtained with posed expressions, though the effect sizes were smaller for the authentic expressions. Third, automatic classification (using LDA classifiers both with and without speaker adaptation) of irritation, resignation, and neutral performed at a level comparable to human performance, though human listeners and machines did not necessarily classify individual utterances similarly. Fourth, clearly perceived exemplars of irritation and resignation were rare in our corpus. These findings were discussed in relation to future research.