Structure reconstructed from motion is usually distorted due to accumulated error and calibration error from extrinsic parameters of video cameras

although the re-projected error sometimes can be small. This paper proposes a novel approach to structure from motion with the geometry conditions existing in the scene

such as the trajectory of a video camera lying on a straight line. With this new method

distortion of the reconstructed results can be removed. We also propose a method to automatically select the optimal coefficients between the original cost and the constraint cost

through which the best reconstruction is obtained. The implemented examples demonstrate very precise structure and motion recovery

which prove the effectiveness and robustness of the proposed method.