Visual-Guided Robot Arm Calibration and Control Using Deep Learning

Project: Research project

Project Details


In industry 4.0 – smart manufacturing, intelligent robot arm plays an important role for manufacturing automation. About “intelligence” of robot arm control, it requires the technologies of computer vision and artificial intelligence – reinforcement (deep) learning in order that robot arm can automatically learn by itself for picking and placing the 3D objects. By considering the cost and maintenance, this work will develop two computer vision technologies: 1) The automatic 3D coordinate calibration between eye-to-hand 2D camera and robot arm base. 2) 2D-visual-guided robot arm control using reinforcement learning for picking and placing the 3D object. Firstly, it is necessary to calibrate the rotation and translation transformation matrix between 3D camera coordinate and 3D robot arm base coordinate when the 2D eye-to-hand camera is applied to precisely guide the robot arm for picking and placing processes. Usually this kind of calibration process takes time and needs professional person to operate. Therefore, this work will develop an automatic calibration technology between 3D coordinates. This calibration process is going to take around 30 seconds with less than 0.5mm precision. Secondly, based on 2D eye-to-hand camera, we will develop a visual-based reinforcement learning technology to detect 3D object and estimate its 3D position. So the robot arm can be guided to operate the picking and placing process precisely. In addition, in order to have higher precision without occlusion problem, the camera will set up at the location, which will be closer to the object, with some pitch angles.
Effective start/end date18-08-0119-07-31


Explore the research topics touched on by this project. These labels are generated based on the underlying awards/grants. Together they form a unique fingerprint.