In this paper, a two-level near-duplicate video detection method based on invariant moment was proposed. To reduce the computational complexity of near-duplicate video detection, a coarse-to-fine approach was adopted in the proposed method. The proposed method is composed of key-frame selection, invariant moment calculation, feature point extraction and matching, similarity measurement, and near-duplicate classifier. After key-frame selection, the proposed method coarsely finds the corresponding frame pairs based on invariant moments. For each chosen frame pair, SURF is used to find the corresponding point pairs between the query frame and the test one. After feature-level, spatial-level, and temporal-level similarity measurement, we can decide whether the query video clip and the test one are near-duplicate. The experimental results show that the proposed method can effectively detect near-duplicate videos. In addition, the proposed method has good performance against possible operations, re-scaling, and frame-rate change.