Aiming at the problem of H.264 inter frame mode selection algorithm

a fast inter frame mode selection algorithm is proposed in this paper. The algorithm selects the macroblock mode by using homogenous analysis of an 8×8 block motion vector

to narrow the scope of the candidate modes and decrease the computational complexity of the H.264 inter frame mode selection algorithm. Meanwhile

the algorithm uses mode correlation and motion vector merging to optimize the UMHexagonS algorithm

which could reduce a large number of motion estimation search points

and obtain higher coding efficiency. The experimental results show that compared with inter frame mode selection algorithm using UMHexagonS

the proposed algorithm can reduce about 50% motion estimation time with only negligible change of PSNR and output stream bit rate.