X-ray image segmentation is an important issue in medical image analysis. Due to inconsistent X-ray absorption, the intensities are usually unevenly distributed and noisy in the processed organ, thus the object segmentation becomes difficult. In this paper we propose a new segmentation method for patella from the lateral knee X-ray images based on the active shape model (ASM). At first, a patella shape model is constructed by principal component analysis (PCA) of corresponding landmarks obtained from a set of training shape. As the knee X-ray image usually contains many anatomical structures, we design a strategy based on edge tracing to place the initial shape model as close to the patella boundary as possible. Then, the shape model is deformed and fitted to the patella boundary by using a dual-optimization approach that includes a genetic algorithm (GA) to get the global geometric transform and ASM to deform the shape model iteratively. Consequently, the proposed method can cope with different knee X-ray images and can segment the patella in an automatic procedure. In the experiment, 20 images were tested and promising results are obtained by the proposed method. This method is found useful for the clinical evaluation and biomechanical study of knee.