SIM-TO-REAL LEARNING OF 2D MULTIPLE SOUND SOURCE LOCALIZATION

• Main Authors: Le Moing, Guillaume Jean Victor Marie, Agravante, Don Joven Ravoy, Vinayavekhin, Phongtharin, Inoue, Tadanobu, Vongkulbhisal, Jayakorn, Munawar, Asim
• Format: Patent
• Language: English
• Published: 02.09.2021
• Subjects: ACOUSTICS; CALCULATING; COMPUTER SYSTEMS BASED ON SPECIFIC COMPUTATIONAL MODELS; COMPUTING; COUNTING; DEAF-AID SETS; ELECTRIC COMMUNICATION TECHNIQUE; ELECTRICITY; LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKEACOUSTIC ELECTROMECHANICAL TRANSDUCERS; MUSICAL INSTRUMENTS; PHYSICS; PUBLIC ADDRESS SYSTEMS; SPEECH ANALYSIS OR SYNTHESIS; SPEECH OR AUDIO CODING OR DECODING; SPEECH OR VOICE PROCESSING; SPEECH RECOGNITION

A computer-implemented method is provided for training a multi-source sound localization model using labeled simulation data and unlabeled real data. The method includes inputting the labeled simulation data and the unlabeled real data respectively into a multi-source sound localization model of a neural network to obtain a localization heatmap from an output layer of the multi-source sound localization model for each of the labeled simulation data and the unlabeled real data. The method further includes inputting the localization heatmap for each of the labeled simulation data and the unlabeled real data into an output discriminator. The method also includes training the output discriminator so that the output discriminator assigns a domain class label to distinguish simulation data from real data. The method additionally includes training, by a hardware process, the multi-source sound localization model by a first adversarial loss for the output discriminator with an original localization model loss.