Rate distortion optimization (RDO) techniques play an important role in high-efficiency video coding (HEVC) because these techniques are usually used to select optimal encoding parameters and implement the trade-off between bit-rate and distortion of reconstructed video. In HEVC

distortion is usually evaluated by mean squared error or sum of absolute difference because they are well-understood quantities and convenient. However

they do not clearly reflect human visual perception because video signals are ultimately received by human eyes. Thus

Lagrangian multiplier must be developed by considering human visual system characteristics. In this paper

a novel RDO algorithm is proposed based on human perception and is applied for inter prediction video coding. First

a human perception factor was defined by the features of human vision system about spatial motion region

texture region

temporal motion activity

and illumination in video. Multiplication of spatial motion region

texture region

temporal motion activity

and illumination factors were used to evaluate human perception. Then

Lagrangian multiplier was adaptively adjusted for each coding tree unit based on human perception factor. Finally

through derivation

we determined that variation of Lagrangian multiplier results in the change of quantization parameters. Hence

quantization parameters were further amended according to the relationship between Lagrangian multiplier and quantization parameters. Experimental results showed that the proposed method improves rate-distortion performance compared with HEVC reference software. For the same structural similarity (SSIM) index

the proposed approach saves bit by 3.1% for random access configuration and 4.9% on average for low delay encoding configuration. Maximal bit rate saving can reach 9.0%. The rate-distortion performance of the proposed approach is better than that of representative literature algorithm. For the same SSIM index

the proposed approach can save the bit by 0.7% for random access configuration and 2.2% for low delay encoding configuration on average. We conducted an experiment to explain the role of each perceptual factor. Experimental results showed that the factor of texture region is better than other factors and is not so good as the combination of four factors. In this paper

we proposed a novel RDO strategy that can adaptively adjust Lagrange multipliers based on different visual characteristics. Experimental results showed that the proposed strategy can improve HEVC coding performance and can achieve bit rate saving with the same reconstructed video quality.