Effective Techniques for Multimodal Data Fusion: A Comparative Analysis

• Published in: Sensors (Basel, Switzerland) Vol. 23; no. 5; p. 2381
• Main Authors: Pawłowski, Maciej, Wróblewska, Anna, Sysko-Romańczuk, Sylwia
• Format: Journal Article
• Language: English
• Published: Switzerland MDPI AG 21.02.2023; MDPI
• Subjects: comparative analysis; data fusion; Data integration; Data processing; Deep learning; deep learning in sensor systems; Electronic data processing; Machine learning; Methods; multimodal learning; multimodal representation; neural networks; Representations; Robotics; Sensors; Poland; France; United States > US; data fusion; comparative analysis; multimodal learning; multimodal representation; deep learning in sensor systems; neural networks
• ISSN: 1424-8220; 1424-8220
• DOI: 10.3390/s23052381

Data processing in robotics is currently challenged by the effective building of multimodal and common representations. Tremendous volumes of raw data are available and their smart management is the core concept of multimodal learning in a new paradigm for data fusion. Although several techniques for building multimodal representations have been proven successful, they have not yet been analyzed and compared in a given production setting. This paper explored three of the most common techniques, (1) the late fusion, (2) the early fusion, and (3) the sketch, and compared them in classification tasks. Our paper explored different types of data (modalities) that could be gathered by sensors serving a wide range of sensor applications. Our experiments were conducted on Amazon Reviews, MovieLens25M, and Movie-Lens1M datasets. Their outcomes allowed us to confirm that the choice of fusion technique for building multimodal representation is crucial to obtain the highest possible model performance resulting from the proper modality combination. Consequently, we designed criteria for choosing this optimal data fusion technique.