A spatio-temporal interest point detection approach in the environment of clutter backgrounds with camera shaking/movements is proposed. It incorporates smoothing functions in the spatial and temporal domain. Non-linear anisotropic diffusion is used in spatial direction to efficiently smooth noises and small structures in cluttered backgrounds and to preserve edge information in human action. The uncertainty of the temporal correlation due to camera shaking/movements is modeled by a suppressing function in proportion to the image gradients. This method efficiently solves the problem of cluttered backgrounds and camera shaking/movements in complex scenes.