Panoramic image generation using deep neural networks

• Published in: Soft computing (Berlin, Germany) Vol. 27; no. 13; pp. 8679 - 8695
• Main Authors: Khamiyev, Izat, Issa, Dias, Akhtar, Zahid, Demirci, M. Fatih
• Format: Journal Article
• Language: English
• Published: Berlin/Heidelberg Springer Berlin Heidelberg 01.07.2023
• Subjects: Artificial Intelligence; Computational Intelligence; Control; Data Analytics and Machine Learning; Engineering; Mathematical Logic and Foundations; Mechatronics; Robotics; Computer vision; Homography matrix; Convolutional neural networks; Image stitching
• ISSN: 1432-7643; 1433-7479
• DOI: 10.1007/s00500-023-08056-5

A traditional approach for panoramic image generation consists of a random sample consensus (RANSAC) algorithm on a set of scale-invariant feature transform (SIFT) correspondences to generate a homography matrix between two images. Although producing adequate results for some type of images, hand-crafted SIFT features are not robust enough for highly varying natural images and the iterative RANSAC algorithm with its randomness does not always find the desired homography matrix. Recently, deep neural networks have been producing significant results in many challenging computer vision problems by learning features from large amounts of data. However, only very few recent works have been applied deep learning to panoramic image generation with the objective of finding feature correspondences and estimating homography matrix. Moreover, the absence of a proper dataset for the image stitching task hinders the standardization of models and comparison of their results. This paper attempts to generate panoramic images by extensively experimenting with various approaches using deep neural networks. The best proposed deep learning model achieved 7.31 and 1.07 pixels of the average absolute value loss for corner difference in X and Y directions, respectively. At the same time, qualitative results demonstrate superiority in comparison with the state-of-the-art SIFT+RANSAC algorithm. Specifically, in 72% of time the proposed framework either produced better results than SIFT+RANSAC or results of the proposed approach and SIFT+RANSAC were indistinguishable. Although SIFT + RANSAC produces better results in 28% of the time with respect to the loss function, our results are still visually comparable in many of these cases. Finally, a novel panoramic image generation dataset is introduced in this paper.