The MAHNOB Mimicry Database: A database of naturalistic human interactions

• Published in: Pattern recognition letters Vol. 66; pp. 52 - 61
• Main Authors: Bilakhia, Sanjay, Petridis, Stavros, Nijholt, Anton, Pantic, Maja
• Format: Journal Article
• Language: English
• Published: Elsevier B.V 15.11.2015
• Subjects: Behavioural mimicry; Motor mimicry; Social signal processing; Temporal modelling; Behavioural mimicry; Motor mimicry; Temporal modelling; Social signal processing
• ISSN: 0167-8655; 1872-7344
• DOI: 10.1016/j.patrec.2015.03.005

•We present an audiovisual dataset for investigation of mimicry behaviour.•We report baseline performances from per-session mimicry classification experiments.•Performance is session-dependent, due to variability in subject expressiveness.•Current mimicry classification methods need more development for spontaneous data. People mimic verbal and nonverbal expressions and behaviour of their counterparts in various social interactions. Research in psychology and social sciences has shown that mimicry has the power to influence social judgment and various social behaviours, including negotiation and debating, courtship, empathy and helping behaviour. Hence, automatic recognition of mimicry behaviour would be a valuable tool in various domains, and especially in negotiation skills enhancement and medical help provision training. In this work, we present the MAHNOB mimicry database, a set of fully synchronised, multi-sensory, audiovisual recordings of naturalistic dyadic interactions, suitable for investigation of mimicry and negotiation behaviour. The database contains 11 h of recordings, split over 54 sessions of dyadic interactions between 12 confederates and their 48 counterparts, being engaged either in a socio-political discussion or negotiating a tenancy agreement. To provide a benchmark for efforts in machine understanding of mimicry behaviour, we report a number of baseline experiments based on visual data only. Specifically, we consider face and head movements, and report on binary classification of video sequences into mimicry and non-mimicry categories based on the following widely-used methodologies: two similarity-based methods (cross correlation and time warping), and a state-of-the-art temporal classifier (Long Short Term Memory Recurrent Neural Network). The best reported results are session-dependent, and affected by the sparsity of positive examples in the data. This suggests that there is much room for improvement upon the reported baseline experiments.