MixedEmotions: An Open-Source Toolbox for Multimodal Emotion Analysis

• Published in: IEEE transactions on multimedia Vol. 20; no. 9; pp. 2454 - 2465
• Main Authors: Buitelaar, Paul, Wood, Ian D., Negi, Sapna, Arcan, Mihael, McCrae, John P., Abele, Andrejs, Robin, Cecile, Andryushechkin, Vladimir, Ziad, Housam, Sagha, Hesam, Schmitt, Maximilian, Schuller, Bjorn W., Sanchez-Rada, J. Fernando, Iglesias, Carlos A., Navarro, Carlos, Giefer, Andreas, Heise, Nicolaus, Masucci, Vincenzo, Danza, Francesco A., Caterino, Ciro, Smrz, Pavel, Hradis, Michal, Povolny, Filip, Klimes, Marek, Matejka, Pavel, Tummarello, Giovanni
• Format: Journal Article
• Language: English
• Published: Piscataway IEEE 01.09.2018; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: affective computing; Audio data; audio processing; Data processing; Embedded systems; Emotion analysis; Emotion recognition; Face recognition; Freeware; Image processing; Linked Data; Media; Open source software; open source toolbox; Speech; text processing; Text recognition; Video; video processing
• ISSN: 1520-9210; 1941-0077
• DOI: 10.1109/TMM.2018.2798287

Recently, there is an increasing tendency to embed functionalities for recognizing emotions from user-generated media content in automated systems such as call-centre operations, recommendations, and assistive technologies, providing richer and more informative user and content profiles. However, to date, adding these functionalities was a tedious, costly, and time-consuming effort, requiring identification and integration of diverse tools with diverse interfaces as required by the use case at hand. The MixedEmotions Toolbox leverages the need for such functionalities by providing tools for text, audio, video, and linked data processing within an easily integrable plug-and-play platform. These functionalities include: 1) for text processing: emotion and sentiment recognition; 2) for audio processing: emotion, age, and gender recognition; 3) for video processing: face detection and tracking, emotion recognition, facial landmark localization, head pose estimation, face alignment, and body pose estimation; and 4) for linked data: knowledge graph integration. Moreover, the MixedEmotions Toolbox is open-source and free. In this paper, we present this toolbox in the context of the existing landscape, and provide a range of detailed benchmarks on standard test-beds showing its state-of-the-art performance. Furthermore, three real-world use cases show its effectiveness, namely, emotion-driven smart TV, call center monitoring, and brand reputation analysis.