H.264/AVC encoding standard effectively improves coding efficiency by using the intra and inter prediction techniques.However

due to the use of variable block sizes and rate-distortion optimization techniques

there is an extremely high computational complexity for inter-prediction in H.264/AVC

which limits the applications of H.264/AVC.This is because the encoder needs to exhaustively check all the prediction modes for identifying the best one.Therefore

reducing the complexity of the encoder is very important especially for real-time applications.Choosing a small number of candidate modes other than all the modes to perform rate-distortion optimization can reduce the complexity and speed-up the encoding process.In this paper

a fast intermode decision algorithm for P-frame encoding based on the decision tree is proposed to reduce the number of modes required to be checked. The inter frame motion estimation for a 16×16 size is firstly performed when a macroblock (MB) of P-frame is encoded by using the inter prediction technique.Additionally

we use the information of residual MB after motion compensation to select a small number of candidate modes from all possible prediction modes to calculate and compare the rate-distortion cost.At first

the number of 4×4 all-zero blocks in the residual MB can describe how accurate the 16×16 inter prediction is.Thus with the statistical analysis of the correlation between the number of 4×4 all-zero blocks in the residual MB and the best prediction mode in full search algorithm

the candidate modes of some MBs can be directly determined based on the number of all-zero blocks in the residual MB.Then

for the remaining MBs

machine-learning tools are used to exploit the correlation between the residual MB information and the best mode in H.264/AVC full search algorithm.By using the decision tree classification algorithm

candidate modes can be selected based on the SATD features of residual MB. The proposed algorithm efficiently reduces the number of candidate modes for P-frame inter-prediction in the H.264/AVC encoder.Experimental results show that the proposed algorithm can achieve significant encoding time saving for all the test video sequences covering a wide range of motion activities. Meanwhile there is the less PSNR degradation and a small bit rate increase

compared with the reference full search algorithm.Although the time-saving of the proposed algorithm is not better than those of the comparative algorithms

the time-savings for video sequences with different motion degrees are almost identical and the rate distortion performance is obvious better. H.264/AVC video coding standard effectively improves coding efficiency but at the cost of high computational complexity.This paper presents a fast and efficient intermode decision algorithm to speed up the encoding process.The algorithm reduces the computational complexity by selecting a small number of candidate modes from the set of all possible prediction modes.We use the decision tree classification algorithm to determine candidate modes based on the information in residual MB after 16×16 inter motion estimation.The experimental results indicate that our algorithm reduces the computational complexity with a little loss in PSNR and increment in the total bit rate.