Automatic inspection on line plays an important role in industrial quality management nowadays. This paper proposes a new computer vision system for automatic steel surface inspection. The system analyzes the images sequentially acquired from steel bar to detect different kinds of defects on the steel surface. Several image processing strategies are used to detect and outline the defects. The detected defects are then classified into different defect types by using a hierarchical neural network classifier. Some manual detection results by field experts are used to verify the correctness of the proposed detection. In defect classification, the results show that the relevance vector machine (RVM) has better accuracy than the back propagation neural network (BPN). The proposed algorithm was found capable of detecting defects on steel surface rapidly and precisely.