Tracking the patella movement trajectory during the bending process of the knee is one essential step to knee pain diagnosis. In order for tracking patella, correct segmentation of the femur and patella from the axial knee MR image is indispensable. But the strong adhesion of the soft tissue around femur and patella, the gray-level similarities of adjacent organs, and the non-uniform gray intensity due to the degradation of the magnetic propagation make the MR image segmentation challenging. In this paper, we proposed a mechanism combining contextual constraint neural network (CCNN) and level set evolution to segment the femur and patella. The segmentation can be divided into two phases. In the first phase SOM and CCNN are applied to extract initial contours of the femur and patella. Consequently in the second phase, modified level set evolution is performed, with the extracted contours as the initial zero level set contour, to accomplish the segmentation of the femur and patella. Our experimental results show that the femur and patella can be correctly segmented for tracking patella movement.