Fisheye camera has been used in many field such as unmanned aerial vehicle, car reversing displays, endoscope, ect. Because the focal length of fisheye camera is too short, it will cause the barrel distortion. So far, most of the barrel distortion correction VLSI architectures are based on polynomial model or division model, and the parameters of polynomial model and division model are usually obtained from the camera parameters, which means that we can only perform barrel distortion correction on images taken by the same camera. So we have implemented the VLSI architecture of this algorithm  which can find out the parameters of polynomial model and division model by images. This algorithm  is only applied to the software, but the VLSI architecture has not been implemented yet. Thus, we are the first one who implement the algorithm into the VLSI architecture. Our design contains 102868 gate counts and works with a clock period of 10 ns and operates at a clock rate of 100 MHz by using TSMC's 90 nm cell library.