A video shot segmentation method based on a two-stage support vector machine (SVM) classifier was proposed. In the first stage，the feature vectors were generated by using the segmented video histogram distance, combining the sliding window with the trap method. Then, the subsegments containing shot boundaries were filtered out by using the SVM classifier. In the second stage，the feature vectors were obtained by histogram distance between different spacing of the frames and time window. The shot detection was implemented by binary tree SVM classification strategy. Experimental results show that the detection results can be improved on both abrupt and gradual shot boundary significantly by the proposed method.