Improving driver’s safety is the main goal of the advanced driver assistance system, which has been widely deployed for proactive driving security in recent years. For road driving, the advanced driver assistance system should visually recognize circular prohibition and triangular warning traffic signs to help drivers to grab complete traffic conditions. In this paper, we proposed a low-computation neural assistance system for traffic sign recognition. First, we proposed shaped-based detection algorithms to detect the regions, which are with circle and triangular traffic signs in designated regions of interest. For classification to those detected regions, we then suggest a convolutional neural network to achieve about 5% improvement of top 1 accuracy compared with LeNet model in German traffic sign recognition benchmarks dataset. For real applications, we also establish a Taiwanese traffic sign database to train the proposed neural network. The simulation results on self-collect driving videos demonstrate that the proposed traffic sign recognition system achieved above 97% recognition rate can be effectively adopted in ADAS applications.