The latest video coding standard high efficiency video coding (HEVC) adopts a more flexible structure and new coding tools compared with other coding standards. The adoption of new technologies results in entirely unrelated prediction residual blocks. An inferior energy concentration can also be obtained by using several methods

such as DCT transform in frequency domain

in less-correlative pixel residual blocks. Among these new coding tools

transform skip mode can enhance the coding efficiency effectively while increasing the coding complexity considerably. Therefore

the difficulty of practical real-time coding is increased. To address the problem

a fast algorithm that reduces the transform skip mode in advance is proposed. The square root of the rate distortion cost is selected as the threshold value by analyzing whether the distribution of the residual block has selected the transform skip modeas the best mode at different rates of distortion costs.To save bits

the HEVC standard defines whether the coefficients of transform units are all zero after transform skip and quantization.The DCT/DST transform is directly chosen as the best mode. A larger quantization parameter can mean a larger percentage of all zero blocks after transform skip mode.Therefore

an exponential model of the rate distortion cost and quantization parameter is established. In the actual coding process

thresholds can be calculated in advance based on the quantization parameters.These thresholds are used to reduce the number of transform units that need to check the transform skip mode. Therefore

the complexity of the transform skip mode is reduced. Only a small number of blocks need to check the transform skip mode

and the thresholds are obtained by offline training.Thus

no additional complexity is added

and the computational complexity of the encoder is reduced. Experimental results show that compared with the standard encoder

the fast algorithm has minimal effect on PSNR and bit rate for standard test sequences that include different scenes. On average

about 70% of the transform units do not need to check the transform skip mode. The exponential model established in this paper fits very well

with a square of determination coefficient larger than 0.95. The algorithm selects the square root of the rate distortion cost as the threshold to reduce the transform skip mode and obtains the threshold by using the exponential model and quantization parameter.The transform skip mode can be reduced in advance. Experimental results indicate that the fast algorithm can reduce the coding complexity effectively because of the addition of transform skip mode with negligible performance loss. The proposed algorithm can be applied to real-time situations. It can reduce the time of transform skip mode significantly and can be further optimized. The relationship between the coding efficiency and coding time can be balanced dynamically by establishing the model of performance loss and pruning percentage in further research.