Adaptive Deep PnP Algorithm for Video Snapshot Compressive Imaging

• Published in: International journal of computer vision Vol. 131; no. 7; pp. 1662 - 1679
• Main Authors: Wu, Zongliang, Yang, Chengshuai, Su, Xiongfei, Yuan, Xin
• Format: Journal Article
• Language: English
• Published: New York Springer US 01.07.2023; Springer; Springer Nature B.V
• Subjects: Adaptive algorithms; Algorithms; Artificial Intelligence; Computer Imaging; Computer Science; Design optimization; Frames (data processing); High speed; Ill posed problems; Image Processing and Computer Vision; Imaging; Machine learning; Noise reduction; Pattern Recognition; Pattern Recognition and Graphics; Reconstruction; S.I. : Physics-Based Vision meets Deep Learning; Special Issue on Physics-Based Vision meets Deep Learning; Video compression; Vision; Deep learning; Deep plug-and-play; Neural networks; Snapshot compressive imaging; Computational imaging
• ISSN: 0920-5691; 1573-1405
• DOI: 10.1007/s11263-023-01777-y

Video Snapshot compressive imaging (SCI) is a promising technique to capture high-speed videos, which transforms the imaging speed from the detector to mask modulating and only needs a single measurement to capture multiple frames. The algorithm to reconstruct high-speed frames from the measurement plays a vital role in SCI. In this paper, we consider the promising reconstruction algorithm framework, namely plug-and-play (PnP), which is flexible to the encoding process comparing with other deep learning networks. One drawback of existing PnP algorithms is that they use a pre-trained denoising network as a plugged prior while the training data of the network might be different from the task in real applications. Towards this end, in this work, we propose the online PnP algorithm which can adaptively update the network’s parameters within the PnP iteration; this makes the denoising network more applicable to the desired data in the SCI reconstruction. Furthermore, for color video imaging, RGB frames need to be recovered from Bayer pattern or named demosaicing in the camera pipeline. To address this challenge, we design a two-stage reconstruction framework to optimize these two coupled ill-posed problems and introduce a deep demosaicing prior specifically for video demosaicing in SCI. Extensive results on both simulation and real datasets verify the superiority of our adaptive deep PnP algorithm. Code is available at https://github.com/xyvirtualgroup/AdaptivePnP_SCI .