The adaptive motion search algorithm and its corresponding architecture for H.264/AVC are proposed in this paper. The correlations of spatial MBs and temporal MB are exploited to predict the search centers. The origin point is chosen as another search center to enhance the motion search in case of prediction error. Moreover, according to the spatial and temporal correlations, three kinds of search patterns are used to reduce the search points. The average number of search points in the proposed algorithm is about 13 and the performance is similar to the full search algorithm. To support the irregular search patterns, a hardware architecture design is also proposed. Synthesis result shows that the total number of gate counts is 21.7k and the maximum operation frequency is 142 MHz. It can achieve the requirement for real-time encoding of 720p-HD video format.