In this paper, we present an efficient chip for the removal of impulse noise in images. Our design uses a mask on each pixel in the image to determine whether it is likely being corrupted by random-valued impulse noise or not. After noise detection, we reconstruct the noisy pixel by considering the possible edges existed in the mask and applying a median filter on it. The memory requirement of the proposed method is quite small. The experimental results demonstrate that our method achieves excellent performance in quantitative evaluation and visual quality. An efficient VLSI architecture for this scheme is developed and it yields a processing rate of about 200 MHz by using TSMC 0.18μm technology.