Impulse noise impacts an image, causing the quality of image to be deteriorated in image transmission or capture. In this paper, we propose a gain factor for the removal of the impulse noise. A 3 × 3 fixed-size local window is employed to analyze each extreme pixel (0 or 255 for an 8-bit gray-level image). All non-extreme pixels are sorted in an ascending order and are grouped according to the variation of pixel levels. If the pixel level between adjacent two sorted pixels varies seriously, a new group is created. Hence, the ratio and median value of each group are computed to determine the values of the gain factors. They are multiplied with the median value of each group to obtain the weighted value which is employed to replace the center pixel with an extreme value, enabling noise-corrupted pixels to be restored. Experimental results show that the proposed method can effectively remove salt-and-pepper noise from a corrupted image for various noise corruption densities (from 10% to 90%); meanwhile, the denoised image is freed from the blurred effect.