In this paper, we propose an approximate square criterion for H.264/AVC intra mode decision. The sum of square difference (SSD) criterion achieves the best video quality but takes much high computation due to the square operation. A sum of approximate square difference (SASD) criterion is proposed to maintain the video quality and reduce the computation. By applying the characteristic of the SSD criterion, simulation results show the ratedistortion performance of the SASD criterion is close to that of the SSD method. For the hardware implementation, synthesized results show that the proposed approximate square operation respectively reduces 75% and 61% in the area cost and timing delay than that of the square function.