Hallucinated-PQA: No reference point cloud quality assessment via injecting pseudo-reference features

• Published in: Expert systems with applications Vol. 243; p. 122953
• Main Authors: Mu, Baoyang, Shao, Feng, Chen, Hangwei, Jiang, Qiuping, Xu, Long, Ho, Yo-Sung
• Format: Journal Article
• Language: English
• Published: Elsevier Ltd 01.06.2024
• Subjects: Hallucination injection; Multi-view features fusion; No reference point cloud quality assessment; Pseudo-reference information; Multi-view features fusion; Hallucination injection; No reference point cloud quality assessment; Pseudo-reference information
• ISSN: 0957-4174; 1873-6793
• DOI: 10.1016/j.eswa.2023.122953

•We propose a novel Hallucinated-PQA to predict the quality of point clouds.•We introduce an image rectification network to generate pseudo-reference images.•Design a hallucination injection block to perceive distortion changes.•Design a multi-scale context fusion module to enhance the feature representation.•Hallucinated-PQA achieves better performance than existing NR and FR metrics. Point clouds (PCs) have been increasingly applied in business and life, but a variety of PC transmission and generation systems inevitably produce various types of distortions. Therefore, it is of great significance to design an objective point cloud quality assessment (PCQA), particularly in a no-reference manner to evaluate the PC systems in the actual situation. However, due to the lack of original PCs, the existing no-reference (NR) PCQA metrics cannot perceive the feature changes caused by distortions, resulting in inaccurate quality prediction. In addition, dependent on the viewing habits of the subjects in the subjective evaluation experiment, a strong contextual correlation among intra- and inter-view local regions at different scales naturally exists, but the existing projection-based no-reference PCQA only using concatenation or average pooling operations cannot reflect this relationship of multi-view features fusion. Considering the above challenges, we propose a novel hallucination-guided NR PCQA framework, namely Hallucinated-PQA. Specifically, we introduce a distortion restoration network to correct multiple projected images in preprocessing to provide pseudo-reference information for NR PCQA. In the feature extraction of distorted PCs, we designed a hallucination injection block (HIB) by utilizing feature differences to assist the feature description of distorted PCs, and a multi-view and multi-scale context fusion (MMCF) module to construct the contextual correlation among intra- and inter-view local regions at different scales. Experimental results show that our Hallucinated-PQA can achieve comparable or better performance than state-of-the-art (SOTA) metrics on four open PCQA databases. The source code will be released at https://github.com/QSBAOYANGMU/Hallucinated-PQA.