CLIP-Llama: A New Approach for Scene Text Recognition with a Pre-Trained Vision-Language Model and a Pre-Trained Language Model

• Published in: Sensors (Basel, Switzerland) Vol. 24; no. 22; p. 7371
• Main Authors: Zhao, Xiaoqing, Xu, Miaomiao, Silamu, Wushour, Li, Yanbing
• Format: Journal Article
• Language: English
• Published: Switzerland MDPI AG 19.11.2024; MDPI
• Subjects: Artificial intelligence; Computer vision; Deep learning; Image retrieval; Language; Llamas; Natural language processing; pre-trained language model; scene text recognition; Semantics; vision-language model; vision-language model; pre-trained language model; scene text recognition
• DOI: 10.3390/s24227371
• ISSN: 1424-8220

This study focuses on Scene Text Recognition (STR), which plays a crucial role in various applications of artificial intelligence such as image retrieval, office automation, and intelligent transportation systems. Currently, pre-trained vision-language models have become the foundation for various downstream tasks. CLIP exhibits robustness in recognizing both regular (horizontal) and irregular (rotated, curved, blurred, or occluded) text in natural images. As research in scene text recognition requires substantial linguistic knowledge, we introduce the pre-trained vision-language model CLIP and the pre-trained language model Llama. Our approach builds upon CLIP’s image and text encoders, featuring two encoder–decoder branches: one visual branch and one cross-modal branch. The visual branch provides initial predictions based on image features, while the cross-modal branch refines these predictions by addressing the differences between image features and textual semantics. We incorporate the large language model Llama2-7B in the cross-modal branch to assist in correcting erroneous predictions generated by the decoder. To fully leverage the potential of both branches, we employ a dual prediction and refinement decoding scheme during inference, resulting in improved accuracy. Experimental results demonstrate that CLIP-Llama achieves state-of-the-art performance on 11 STR benchmark tests, showcasing its robust capabilities. We firmly believe that CLIP-Llama lays a solid and straightforward foundation for future research in scene text recognition based on vision-language models.