VCGAN: Video Colorization With Hybrid Generative Adversarial Network

• Published in: IEEE transactions on multimedia Vol. 25; pp. 3017 - 3032
• Main Authors: Zhao, Yuzhi, Po, Lai-Man, Yu, Wing-Yin, Rehman, Yasar Abbas Ur, Liu, Mengyang, Zhang, Yujia, Ou, Weifeng
• Format: Journal Article
• Language: English
• Published: Piscataway IEEE 2023; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Color; Colorization; Consistency; Feature extraction; Frames (data processing); Generative adversarial networks; Gray scale; Image color analysis; placeholder feature extractor; Semantics; Spatiotemporal phenomena; Task analysis; Training; video colorization
• ISSN: 1520-9210; 1941-0077
• DOI: 10.1109/TMM.2022.3154600

We propose a Video Colorization with Hybrid Generative Adversarial Network (VCGAN), an improved approach to video colorization using end-to-end learning and recurrent architecture. The VCGAN addresses two prevalent issues in the video colorization domain: Temporal consistency and the unification of colorization network and refinement network into a single architecture. To enhance colorization quality and spatiotemporal consistency, the mainstream of the generator in VCGAN is assisted by two additional networks, i.e., global feature extractor and placeholder feature extractor, respectively. The global feature extractor encodes the global semantics of grayscale input to enhance colorization quality, whereas the placeholder feature extractor serves as a feedback connection to encode the semantics of the previous colorized frame in order to maintain spatiotemporal consistency. If changing the input for placeholder feature extractor as grayscale input, the hybrid VCGAN also has the potential to colorize single images. To improve the color consistency of far frames, we propose a dense long-term loss that minimizes the temporal disparity of every two remote frames. Trained with colorization and temporal losses jointly, VCGAN strikes a good balance between video color vividness and spatiotemporal continuity. Experimental results demonstrate that VCGAN produces higher-quality and temporally more consistent colorful videos than existing approaches.