It is important to organize the unstructured video data properly for content video analysis and application. However

existing shot-based video analysis can not get good performance for effective browsing and retrieval because shots are too numerous to handle and can not convey meaningful semantics

so it is necessary to organize video content with the scene structure

which is a more meaningful and high-level semantic video unit. In this paper

a simple and efficient method is presented for video scene detection. As the buildup of video scene should obey the film grammar

it concludes the types of the scenes that are widely used in filmmaking firstly. The rules to generating the video scene are put out then. Based on the rules

a scheme for video scene detection is proposed: The video stream is segmented into shots through shot boundary detection; Key frame extraction is performed based on the content variation of shots; A new clustering method based on the sliding shot window is used to group the shots into shot clusters; A correlation function between shot clusters is defined to analysis the correlations of them and to construct the final video scene structure. Experimental results verify it is an efficient method.