Detecting human in crowded environment is profitable but challenging in video surveillance. We propose an efficient human detection method by combining both motion and appearance clues. Moving pixels are first extracted by background subtraction, and then a filtering step is used to narrow the range for human template matching. We utilize integral images to fast generate shape information from edge maps of each frame and define the matching probability to be capable of detecting both full-body and partial-body. Representative human templates are constructed by sparse contours on the basis of the point distribution model (PDM). Moreover, linear regression analysis is also applied to adaptively adjust the template sizes. With the aid of the proposed foreground ratio filtering and the multi-sized template matching techniques, our method not only can efficiently detect human in a crowded environment but also largely enhance the detection accuracy.