Multi-Temporal Ultra Dense Memory Network for Video Super-Resolution

• Published in: IEEE transactions on circuits and systems for video technology Vol. 30; no. 8; pp. 2503 - 2516
• Main Authors: Yi, Peng, Wang, Zhongyuan, Jiang, Kui, Shao, Zhenfeng, Ma, Jiayi
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.08.2020; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Artificial neural networks; Convolutional neural network (CNN); Correlation; Data integration; Feature extraction; Feature maps; Frames (data processing); Image reconstruction; Motion estimation; multi-temporal fusion; Neural networks; ultra dense memory block (UDRB); video super-resolution
• ISSN: 1051-8215; 1558-2205
• DOI: 10.1109/TCSVT.2019.2925844

Video super-resolution (SR) aims to reconstruct the corresponding high-resolution (HR) frames from consecutive low-resolution (LR) frames. It is crucial for video SR to harness both inter-frame temporal correlations and intra-frame spatial correlations among frames. Previous video SR methods based on convolutional neural network (CNN) mostly adopt a single-channel structure and a single memory module, so they are unable to fully exploit inter-frame temporal correlations specific for video. To this end, this paper proposes a multi-temporal ultra-dense memory (MTUDM) network for video super-resolution. Particularly, we embed convolutional long-short-term memory (ConvLSTM) into ultra-dense residual block (UDRB) to construct an ultra-dense memory block (UDMB) for extracting and retaining spatio-temporal correlations. This design also reduces the layer depth by expanding the width, thus avoiding training difficulties, such as gradient exploding and vanishing under a large model. We further adopt multi-temporal information fusion (MTIF) strategy to merge the extracted temporal feature maps in consecutive frames, improving the accuracy without requiring much extra computational cost. The experimental results on extensive public datasets demonstrate that our method outperforms the state-of-the-art methods by a large margin.