Currently, the conventional approach for the hot rolled steel defect inspections and decisions are made manually based on the standard guideline. However, the quality of the assessments is not consistent due to human error. Therefore, it is essential to develop an automatic inspection system that can decide whether to hold or release the rusted hot rolled steel based on its severity. Artificial intelligence has recently become one of the most frequently cited topics in the development of product defects inspection systems. This project aims to develop a deep learning-based hot rolled steels rust detection system to assist decision-making. The detection processes can be divided into hot rolled steel detection, rust detection, and the decision-making process. Object detection and color detection techniques are adopted in the model for hot rolled steel detection and rust detection respectively. In this study, there are three types of deep learning object detection framework were tested which are Single Shot Detector (SSD) MobileNetv1, SSD MobileNetv2 and Faster RCNN. As Single Shot Detector (SSD) MobileNetv2 has the optimum performance in terms of accuracy and inference speed, it was chosen as the deep learning architecture for the object detection while Hue Saturation Value (HSV) color model is used for the color detection. The hold/release decision-making output is based on the percentage of the detected rusty area from the image. Based on the simulation results, the accuracy of the rust defect detection for hold and release are 96.05% and 97.92%, respectively.