A fast intermode decision algorithm based on analysis of inter prediction residual

• Published in: 2014 IEEE 16th International Workshop on Multimedia Signal Processing (MMSP) pp. 1 - 4
• Main Authors: Luheng Jia, Au, Oscar C., Chi-ying Tsui, Wei Dai, Pengfei Wan
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.09.2014
• Subjects: Algorithm design and analysis; Complexity theory; Encoding; Motion estimation; Quantization (signal); Signal processing algorithms; Video coding
• DOI: 10.1109/MMSP.2014.6958827

Rate-distortion-optimized (RDO) intermode decision is one of the most effective tools that greatly improves the coding performance of modern coding standards, for example, H.264/AVC and HEVC. However, RDO intermode decision also leads to extremely intense computation. To reduce the complexity, a fast intermode decision algorithm is presented in this paper. Mathematical analysis of inter prediction residual is performed which explicitly shows the impact of edge information and motion characteristics to the prediction accuracy. Moreover, It is shown that with fixed quantization step that minimizing of R-D costs over different partition types is equivalent to minimizing the variance of transformed residual which can be expressed by motion vector and edge gradient components. In consequence, the complex calculation of rate and distortion is replaced by simple pre-analysis of video content. The repetition of motion estimation (ME) and entropy coding process are avoided. Inspired by the theoretical analysis, a fast inter mode decision algorithm is proposed. Experimental results show that the fast method achieves considerable complexity reduction with negligible coding performance degradation.