In this paper a novel Multi-layer Homography algorithm for background modeling under free moving camera environment is proposed. Background is composed of many planes. RANSAC is used to find these different planes

called Multi-Layer Homography. Each pixel definitely belongs to certain plane. Transformed by the corresponding Homography

each pixel in each frame can find its match in the subsequent frame if it occurs in the shared view of these two frames. Thus

background model can be built. Experiment shows it is effective for background modeling under free moving camera environments.