The problem of virtualized reality

or view synthesis of real scenes

has received increasing attention in recent years. This paper presents a systematic approach to automatically construct the 3D natural scene from video sequences captured by a vibrated camera with only approximately known motion. A two stage method is presented for the 3D panoramic scene modeling which consists of (1) image stabilization by motion filtering and (2) depth estimation and depth boundary localization. The two stage method not only combines Zheng and Tsuji's panoramic image method with Baker's epipolar plane image analysis

resulting the so called panoramic epipolar plane image method

but also generalizes them to handle image sequence vibrations due to the uncontrollable fluctuation of the camera. The two stage method bypasses the correspondence problem and ill posed problem encountered in the general motion analysis

and avoids the local minimum problem in the spatial constraint based iteration method. This paper mainly describes the robust algorithms of spatio temporal orientation detection of spatio temporal textures

boundary localization of kinetic occlusion and depth layering of the panoramic scene. Two kinds of depth layered representations

depth layered region and depth layered setting

have been developed

which found a ground base for landmark selection of global localization and image synthesis of virtualized reality.