Current screening of mammography results in a high recall rate. Furthermore, distinguishing between BI-RADS 3 and BI-RADS 4 is a challenge for radiologists. In order to help radiologists' diagnosis, researches of CAD system recently have shown that methods of deep learning can significantly improve lesion detection, segmentation, and classification. However, there is not enough evidence to show that deep learning models can reduce the high recall rate because few researches provide the performance of cases in BI-RADS 3 and BI-RADS 4. Moreover, few researches extended the current models to involve images in CC and MLO in a single prediction. Thus, we proposed convolutional neural networks to classify breast cancer. Our model could predict images in four input sizes. Besides, we extended our model to consider images in CC and MLO in a single prediction. To validate our models, we split the data depending on patients rather than images. Our training set was composed of 4255 images, and test set contained 355 images that were proven by biopsy and callback. The overall performance of human experts yielded on an accuracy of 65.3% while our model achieved a better accuracy of 79.6%. Besides, the performance of cases in BI-RADS 3 and 4 by human experts was accuracy of 54.1%, but our model maintained a high accuracy of 75.7%. When we combined images in CC and MLO in the single prediction, we achieved AUC of 0.86.