MF-LPR2: Multi-frame license plate image restoration and recognition using optical flow

• Published in: Computer vision and image understanding Vol. 256
• Main Authors: Na, Kihyun, Oh, Junseok, Cho, Youngkwan, Kim, Bumjin, Cho, Sungmin, Choi, Jinyoung, Kim, Injung
• Format: Journal Article
• Language: English
• Published: Elsevier Inc 01.05.2025
• Subjects: License plate image dataset; License plate recognition; Multi-frame image restoration; Optical flow; Super resolution; Multi-frame image restoration; License plate image dataset; License plate recognition; Super resolution; Optical flow
• ISSN: 1077-3142
• DOI: 10.1016/j.cviu.2025.104361

License plate recognition (LPR) is important for traffic law enforcement, crime investigation, and surveillance. However, license plate areas in dash cam images often suffer from low resolution, motion blur, and glare, which make accurate recognition challenging. Existing generative models that rely on pretrained priors cannot reliably restore such poor quality images, frequently introducing severe artifacts and distortions. To address this issue, we propose a novel multi-frame license plate restoration and recognition framework, MF-LPR2, which addresses ambiguities in poor quality images by aligning and aggregating neighboring frames instead of relying on pretrained knowledge. To achieve accurate frame alignment, we employ a state-of-the-art optical flow estimator in conjunction with carefully designed algorithms that detect and correct erroneous optical flow estimations by leveraging the spatio-temporal consistency inherent in license plate image sequences. Our approach enhances both image quality and recognition accuracy while preserving the evidential content of the input images. In addition, we constructed a novel Realistic LPR (RLPR) dataset to evaluate MF-LPR2. The RLPR dataset contains 200 pairs of low-quality license plate image sequences and high-quality pseudo ground-truth images, reflecting the complexities of real-world scenarios. In experiments, MF-LPR2 outperformed eight recent restoration models in terms of PSNR, SSIM, and LPIPS by significant margins. In recognition, MF-LPR2 achieved an accuracy of 86.44%, outperforming both the best single-frame LPR (16.18%) and the multi-frame LPR (82.55%) among the eleven baseline models. The results of ablation studies confirm that our filtering and refinement algorithms significantly contribute to these improvements. •Propose MF-LPR2, a multi-frame framework to restore low-quality license plates.•Align frames via novel optical flow filtering and refinement for robust restoration.•Introduces PDNF-k metric to measure spurious artifacts in restoration outputs.•Presents RLPR dataset: 200 dash cam sequences with 31 frames each for testing.•Achieve notable improvement over baselines in image quality and recognition accuracy.