Spatio-Temporal Detail Information Retrieval for Compressed Video Quality Enhancement

The past few years have witnessed the great success of multi-frame quality enhancement for compressed video. Although the existing methods based on deformable alignment have achieved the state-of-the-art performance, they do not pay enough attention to the recovery of detail information. In this wor...

Full description

Saved in:
Bibliographic Details
Published inIEEE transactions on multimedia Vol. 25; pp. 1 - 14
Main Authors Luo, Dengyan, Ye, Mao, Li, Shuai, Zhu, Ce, Li, Xue
Format Journal Article
LanguageEnglish
Published Piscataway IEEE 01.01.2023
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subjects
Alignment
compressed video
Convolution
Correlation coefficients
deep learning
deformable convolution
Encoding
Formability
Image coding
Information retrieval
Kernel
Modules
multi-frame
Offsets
Optical flow
Quality enhancement
Recovery
Spatial data
State of the art
Three-dimensional displays
Video compression
Visualization
Online AccessGet full text

Cover

More Information
Summary:The past few years have witnessed the great success of multi-frame quality enhancement for compressed video. Although the existing methods based on deformable alignment have achieved the state-of-the-art performance, they do not pay enough attention to the recovery of detail information. In this work, we propose a Spatio-Temporal Detail Retrieval (STDR) method to promote the recovery of detail information. To alleviate the problem of inaccurate deformable offsets caused by the fixed receptive field, motivated by multi-task learning, we design a plug-and-play Multi-path Deformable Alignment (MDA) module to generate more accurate offsets by integrating the alignment features of different receptive fields, so that the temporal detail information can be better recovered. For the spatial detail information restoration, several residual dense blocks with channel attention layer are utilized in the reconstruction module to explore valuable high-frequency spatial information from the fused multi-path alignment features. Meanwhile, a complementary loss function based on the Pearson correlation coefficient is developed to ameliorate the over-smoothing shortcoming caused by pixel-wise mean square or absolute value loss. Experimental results demonstrate that the proposed STDR network achieves superior performance compared with the state-of-the-art methods in both quantitative and qualitative evaluations.
ISSN:1520-9210
1941-0077
DOI:10.1109/TMM.2022.3214775