The rate distortion optimization technique suggested in H.264/AVC searches the best coding mode consumed extremely high computation complexity in intra and inter predictions due to full mode search. This paper presents a fast direction detection algorithm to effectively reduce the number of the possible candidate modes for H.264 intra prediction. The proposed fast algorithm is based on the direction detection of the edge inside the block. Instead of full intra mode search, the proposed method only searches 4 modes for 4x4 luminance blocks, and 2 modes for 16x16 luminance and 8x8 chrominance blocks. Experimental results reveal that the proposed algorithm significantly reduces the encoding time over 60% in average with slight loss in bit-rate and PSNR.