In this paper

the similarities of bits

reference frame selection and coded block pattern(CBP) of a coding macroblock(MB) between each inter mode and optimal mode have been investigated. The bits and CBP’s distribution characteristics of a MB which encoded by the optimal mode have been analyzed

and the result shows that the coding modes can be differentiated by bits and CBP. Based on these characteristics

a novel low complexity rate distortion optimization (RDO) algorithm which combination reference frame selection with mode selection RDO is proposed. This algorithm utilizes P16×16 mode’s reference frame to select the candidate reference frames for other inter coding modes

and the candidate modes are selected by bits and CBP of mode P16×16. The optimal mode will be selected in all candidate reference frames and candidate modes with RDO. The algorithm is evaluated using a wide range of test sequences of different classes. Comparing with RDO in JM12.0

the experiments results demonstrate a reduction coding time 71.20% in average with 1.32% of bits increase and 0.0752dB of Luma PSNR loss.