Analysis of multimodal signals using redundant representations

• Published in: IEEE International Conference on Image Processing 2005 Vol. 3; pp. 46 - 49
• Main Authors: Monaci, G., Divorra Escoda, O., Vandergheynst, P.
• Format: Conference Proceeding
• Language: English
• Published: IEEE 2005
• Subjects: Cepstral analysis; Dictionaries; Fourier transforms; Matching pursuit algorithms; Mutual information; Physics; Signal analysis; Signal processing; Signal processing algorithms; Signal representations
• ISBN: 9780780391345; 0780391349
• ISSN: 1522-4880; 2381-8549
• DOI: 10.1109/ICIP.2005.1530349

In this work we explore the potentialities of a framework for the representation of audio-visual signals using decompositions on overcomplete dictionaries. Redundant decompositions may describe audio-visual sequences in a concise fashion, preserving good representation properties thanks to the use of redundant, well designed, dictionaries. We expect that this helps us overcome two typical problems of multimodal fusion algorithms. On one hand, classical representation techniques, like pixel-based measures (for the video) or Fourier-like transforms (for the audio), take into account only marginally the physics of the problem. On the other hand, the input signals have large dimensionality. The results we obtain by making use of sparse decompositions of audio-visual signals over redundant codebooks are encouraging and show the potentialities of the proposed approach to multimodal signal representation.