DeepCloud: Ground-Based Cloud Image Categorization Using Deep Convolutional Features

• Published in: IEEE transactions on geoscience and remote sensing Vol. 55; no. 10; pp. 5729 - 5740
• Main Authors: Ye, Liang, Cao, Zhiguo, Xiao, Yang
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.10.2017; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Aggregation; Artificial neural networks; Classification; Cloud computing; Clouds; Computer vision; Convolutional codes; Convolutional neural network (CNN); Data mining; deep learning; Encoding; Feature extraction; Fisher vector (FV); ground-based cloud image categorization; Handicrafts; Histograms; Image classification; Mapping; Meteorology; Methods; Neural networks; Pattern analysis; pattern mining; Semantics; State of the art; Visualization
• ISSN: 0196-2892; 1558-0644
• DOI: 10.1109/TGRS.2017.2712809

Accurate ground-based cloud image categorization is a critical but challenging task that has not been well addressed. One of the essential issues that affect the performance is to extract the representative visual features. Nearly all of the existing methods rely on the hand-crafted descriptors (e.g., local binary patterns, CENsus TRsansform hISTogram, and scale-invariant feature transform). Their limited discriminative power indeed leads to the unsatisfactory performance. To alleviate this, we propose "DeepCloud" as a novel cloud image feature extraction approach by resorting to the deep convolutional visual features. In the recent years, the deep convolutional neural network (CNN) has achieved the promising results in lots of computer vision and image understanding fields. Nevertheless, it has not been applied to cloud image classification yet. Thus, we actually pay the first effort to fill this blank. Since cloud image classification can be attributed to a multi-instance learning problem, simply employing the convolutional features within CNN cannot achieve the promising result. To address this, Fisher vector encoding is applied to executing the spatial feature aggregation and high-dimensional feature mapping on the raw deep convolutional features. Moreover, the hierarchical convolutional layers are used simultaneously to capture the fine textural characteristics and high-level semantic information in the unified manner. To further leverage the performance, a cloud pattern mining and selection method are also proposed. It targets at finding the discriminative local patterns to better distinguish the different kinds of clouds. The experiments on a challenging ground-based cloud image data set demonstrate the superiority of the proposition over the state-of-the-art methods.