Palpation is an important procedure for surgeons to diagnose pathology of soft tissues of internal organs such as the liver and the stomach. A vision-based tactile sensor system to aid the surgeons in diagnosis of soft tissues was developed. The sensor was consisted of a CCD, a focusing lens, and a spherical shell sensor head made of transparent PDMS. The CCD can quantify the contact area between the shell sensor and the indented soft tissue. Finite element analyses of the contact mechanics between the sensor head and soft tissues were performed to determine the optimal geometry and elastic properties of the sensor head. After calibrations, an empirical formula for estimating stiffness of soft tissues was obtained and the sensor was employed to measure in vitro Young's moduli of porcine livers and boiled porcine livers that simulated the hepatocirrhosis. The results showed that the vision-based tactile sensor system is quite linear and has a sensitivity of 12.08 (g/mm2). The tactile sensor succeeds in differentiating the pathological from the normal liver tissues.