Multi-modal data processing method and device, electronic equipment and storage medium

• Main Author: ZHANG YUNXUAN
• Format: Patent
• Language: Chinese; English
• Published: 26.01.2024
• Subjects: CALCULATING; COMPUTING; COUNTING; ELECTRIC DIGITAL DATA PROCESSING; PHYSICS

The invention provides a multi-modal data processing method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring multi-modal data included in input content; feature extraction processing is carried out based on the multi-modal data, sub-modal features corresponding to each modal are obtained, and the dimensions of different sub-modal features are different; performing causal relationship conversion on the sub-modal feature with the highest dimension in the plurality of sub-modal features to obtain a causal feature vector, the causal feature vector having the same dimension as other sub-modal features, and the other sub-modal features being the sub-modal features except the sub-modal feature with the highest dimension in the plurality of sub-modal features; splicing the causal feature vector with other sub-modal features to obtain a spliced feature sequence; and generating reply content of the input content based on the spliced feature sequence. According t