A MLP architecture fusing RGB and CASSI for computational spectral imaging

• Published in: Computer vision and image understanding Vol. 249; p. 104214
• Main Authors: Cai, Zeyu, Hong, Ru, Lin, Xun, Yang, Jiming, Ni, YouLiang, Liu, Zhen, Jin, Chengqian, Da, Feipeng
• Format: Journal Article
• Language: English
• Published: Elsevier Inc 01.12.2024
• Subjects: CASSI; Compressive sensing; Knowledge-transfer; MLP; Snapshot; MLP; Snapshot; Compressive sensing; CASSI; Knowledge-transfer
• ISSN: 1077-3142
• DOI: 10.1016/j.cviu.2024.104214

The coded Aperture Snapshot Spectral Imaging (CASSI) system offers significant advantages in dynamically acquiring hyper-spectral images compared to traditional measurement methods. However, it faces the following challenges: (1) Traditional masks rely on random patterns or analytical design, limiting CASSI’s performance improvement. (2) Existing CASSI reconstruction algorithms do not fully utilize RGB information. (3) High-quality reconstruction algorithms are often slow and limited to offline scene reconstruction. To address these issues, this paper proposes a new MLP architecture, Spectral–Spatial MLP (SSMLP), which replaces the transformer structure with a network using CASSI measurements and RGB as multimodal inputs. This maintains reconstruction quality while significantly improving reconstruction speed. Additionally, we constructed a teacher-student network (SSMLP with a teacher, SSMLP-WT) to transfer the knowledge learned from a large model to a smaller network, further enhancing the smaller network’s accuracy. Extensive experiments show that SSMLP matches the performance of transformer-based structures in spectral image reconstruction while improving inference speed by at least 50%. The reconstruction quality of SSMLP-WT is further improved by knowledge transfer without changing the network, and the teacher boosts the performance by 0.92 dB (44.73 dB vs. 43.81 dB). •Fusing RGB and CASSI to improve the quality of reconstruction.•A novel MLP network for hyperspectral reconstruction with faster speed.•Knowledge Transfer via a teacher-student framework improves accuracy.•Extensive experiments demonstrate the effectiveness of the proposed method.