Real-Time Controllable Denoising for Image and Video

• Published in: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 14028 - 14038
• Main Authors: Zhang, Zhaoyang, Jiang, Yitong, Shao, Wenqi, Wang, Xiaogang, Luo, Ping, Lin, Kaimo, Gu, Jinwei
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.06.2023
• Subjects: Interpolation; Low-level vision; Noise reduction; Pipelines; Process control; Streaming media; Training; User interfaces
• ISSN: 2575-7075
• DOI: 10.1109/CVPR52729.2023.01348

Controllable image denoising aims to generate clean samples with human perceptual priors and balance sharpness and smoothness. In traditional filter-based denoising methods, this can be easily achieved by adjusting the filtering strength. However, for NN (Neural Network)-based models, adjusting the final denoising strength requires performing network inference each time, making it almost impossible for real-time user interaction. In this paper, we introduce Real-time Controllable Denoising (RCD), the first deep image and video denoising pipeline that provides a fully controllable user interface to edit arbitrary denoising levels in real-time with only one-time network inference. Unlike existing controllable denoising methods that require multiple denoisers and training stages, RCD replaces the last output layer (which usually outputs a single noise map) of an existing CNN-based model with a lightweight module that outputs multiple noise maps. We propose a novel Noise Decorrelation process to enforce the orthogonality of the noise feature maps, allowing arbitrary noise level control through noise map interpolation. This process is network-free and does not require network inference. Our experiments show that RCD can enable real-time editable image and video denoising for various existing heavy-weight models without sacrificing their original performance.