Fish detection and species classification in underwater environments using deep learning with temporal information

• Published in: Ecological informatics Vol. 57; p. 101088
• Main Authors: Jalal, Ahsan, Salman, Ahmad, Mian, Ajmal, Shortis, Mark, Shafait, Faisal
• Format: Journal Article
• Language: English
• Published: Elsevier B.V 01.05.2020
• Subjects: Automatic fish detection; Biomass estimation; camouflage; data collection; Deep learning; fish; Fish sampling; Fish species classification; Gaussian mixture models; Optical flow; Underwater video imagery; Western Australia; Western Australia; Deep learning; Automatic fish detection; Biomass estimation; Gaussian mixture models; Fish species classification; Fish sampling; Underwater video imagery; Optical flow
• ISSN: 1574-9541
• DOI: 10.1016/j.ecoinf.2020.101088

It is important for marine scientists and conservationists to frequently estimate the relative abundance of fish species in their habitats and monitor changes in their populations. As opposed to laborious manual sampling, various automatic computer-based fish sampling solutions in underwater videos have been presented. However, an optimal solution for automatic fish detection and species classification does not exist. This is mainly because of the challenges present in underwater videos due to environmental variations in luminosity, fish camouflage, dynamic backgrounds, water murkiness, low resolution, shape deformations of swimming fish, and subtle variations between some fish species. To overcome these challenges, we propose a hybrid solution to combine optical flow and Gaussian mixture models with YOLO deep neural network, an unified approach to detect and classify fish in unconstrained underwater videos. YOLO based object detection system are originally employed to capture only the static and clearly visible fish instances. We eliminate this limitation of YOLO to enable it to detect freely moving fish, camouflaged in the background, using temporal information acquired via Gaussian mixture models and optical flow. We evaluated the proposed system on two underwater video datasets i.e., the LifeCLEF 2015 benchmark from the Fish4Knowledge repository and a dataset collected by The University of Western Australia (UWA). We achieve fish detection F-scores of 95.47% and 91.2%, while fish species classification accuracies of 91.64% and 79.8% on both datasets respectively. To our knowledge, these are the best reported results on these datasets, which show the effectiveness of our proposed approach. •Automatic fish sampling for rapid estimation of fish biomass is gaining interest of marine scientists.•Conventional video processing approaches fail to yield favourable results using underwater videos having extreme variation.•Recently, deep neural networks have been used to achieve state-of-the-art results in various fish-related tasks.•We propose to improve the deep neural networks using temporal information with probabilistic modelling and optical flow.•Our results on benchmark datasets show significant improvement over the current algorithms.