Multi-Modal Dynamic Fusion for Defect Detection in Electronic Products: A Novel Approach Based on Energy and Deep Learning

• Published in: IEEE access Vol. 13; pp. 118565 - 118573
• Main Authors: Liu, Yulin, Gao, Yang
• Format: Journal Article
• Language: English
• Published: Piscataway IEEE 2025; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Accuracy; Artificial neural networks; Convolutional neural networks; Data analysis; Data models; Defect detection; Defects; Image sensors; Machine learning; Manufacturing; Multi-modal fusion; Multilayer perceptrons; Multilayers; Performance measurement; Quality assurance; quality control; Quality standards; Sensor phenomena and characterization; Sensors; Soft sensors; transformer; Transformers; Uncertainty
• ISSN: 2169-3536; 2169-3536
• DOI: 10.1109/ACCESS.2025.3584551

As electronic products continue to evolve in complexity, maintaining stringent quality standards during manufacturing presents mounting challenges. Conventional defect detection approaches, which typically depend on a single modality, often fall short in both efficiency and reliability. To address these shortcomings, this study introduces a dynamic multi-modal fusion framework that leverages data from sensors, visual imagery, and component attributes to enhance detection performance. Specifically, Transformer architectures are employed for sensor data analysis, Convolutional Neural Networks (CNNs) are applied to process image data, and Multi-Layer Perceptrons (MLPs) are used to represent part-level features. A distinguishing element of this approach is an energy-based late-stage fusion mechanism that adaptively modulates each modality's influence according to its uncertainty level. Empirical evaluations demonstrate that the proposed model achieves superior results across multiple performance metrics-including accuracy, precision, recall, and F1 score-compared to conventional and unimodal systems. These findings underscore the model's potential in advancing practical defect detection and quality assurance in manufacturing environments.