Side information-driven image coding for hybrid machine–human vision

• Published in: EURASIP journal on image and video processing Vol. 2025; no. 1; pp. 3 - 24
• Main Authors: Zhang, Zhongpeng, Liu, Ying, Peng, Wen-Hsiao
• Format: Journal Article
• Language: English
• Published: Cham Springer International Publishing 28.01.2025; Springer Nature B.V; SpringerOpen
• Subjects: Biometrics; Codec; Engineering; Image classification; Image coding; Image coding for machines; Image compression; Image Processing and Computer Vision; Image reconstruction; Image segmentation; Image transmission; Information systems; Machine learning; Machine vision; Pattern Recognition; Semantic segmentation; Semantics; Side information; Signal,Image and Speech Processing; Vision systems; Visual coding for humans and machines; Image compression; Image coding for machines; Side information; Semantic segmentation; Image classification
• ISSN: 1687-5281; 1687-5176; 1687-5281
• DOI: 10.1186/s13640-024-00661-0

With the development of machine learning, advanced photography and image transmission systems, images are being processed more and more by machines, so image coding for machines (ICM) came into being. After the image codec compresses and transmits the image, the image will be handed over to machine vision task networks. These vision tasks include image classification, semantic segmentation, and so on. We propose a side information-driven image coding for hybrid machine–human vision (SICMH) framework, not only for machine vision tasks, but also for human vision-oriented image reconstruction. The proposed SICMH framework can perform image classification, semantic segmentation, and coarse image reconstruction by using purely the side information. Moreover, SICMH can perform fine image reconstruction by using the residue information. In particular, we propose a multi-scale feature fusion block to enhance the usage of side information, and a novel semantic segmentation network named modified TrSeg to generate better semantic segmentation maps. The experimental results well demonstrated the effectiveness of our proposed framework. SICMH achieves the same image classification and semantic segmentation accuracy as the existing traditional or learning-based multi-task ICM frameworks using the lowest bitrate. For the image reconstruction task, the proposed SICMH achieved the same PSNR as existing learning-based multi-task hybrid ICM frameworks and the traditional image codec BPG again with the lowest bitrate.