MobileSal: Extremely Efficient RGB-D Salient Object Detection

• Published in: IEEE transactions on pattern analysis and machine intelligence Vol. 44; no. 12; pp. 10261 - 10269
• Main Authors: Wu, Yu-Huan, Liu, Yun, Xu, Jun, Bian, Jia-Wang, Gu, Yu-Chao, Cheng, Ming-Ming
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.12.2022; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Color imagery; Convolution; efficiency; Feature extraction; Fuses; Image restoration; implicit depth restoration; Neural networks; Object detection; Object recognition; Representations; RGB-D salient object detection; Salience; Semantics; Streaming media
• ISSN: 0162-8828; 2160-9292; 1939-3539
• DOI: 10.1109/TPAMI.2021.3134684

The high computational cost of neural networks has prevented recent successes in RGB-D salient object detection (SOD) from benefiting real-world applications. Hence, this article introduces a novel network, MobileSal, which focuses on efficient RGB-D SOD using mobile networks for deep feature extraction. However, mobile networks are less powerful in feature representation than cumbersome networks. To this end, we observe that the depth information of color images can strengthen the feature representation related to SOD if leveraged properly. Therefore, we propose an implicit depth restoration (IDR) technique to strengthen the mobile networks' feature representation capability for RGB-D SOD. IDR is only adopted in the training phase and is omitted during testing, so it is computationally free. Besides, we propose compact pyramid refinement (CPR) for efficient multi-level feature aggregation to derive salient objects with clear boundaries. With IDR and CPR incorporated, MobileSal performs favorably against state-of-the-art methods on six challenging RGB-D SOD datasets with much faster speed (450fps for the input size of <inline-formula><tex-math notation="LaTeX">320\times 320</tex-math> <mml:math><mml:mrow><mml:mn>320</mml:mn><mml:mo>×</mml:mo><mml:mn>320</mml:mn></mml:mrow></mml:math><inline-graphic xlink:href="cheng-ieq1-3134684.gif"/> </inline-formula>) and fewer parameters (6.5M). The code is released at https://mmcheng.net/mobilesal .