In this paper, we propose a bean defect picking system with the quad-partitioning-based robotic arm guidance method, aimed at automatically and precisely picking bean defects in coffee industry. We assume the adopted inexpensive devices, including a robotic arm, a camera, and an IoT (Internet of Things) device, have only basic functions. For successfully picking the small size of beans as possible, stably moving the arm head to the target bean is the key technique in this topic. To achieve this goal under hardware limits, we design an iterative robotic arm guidance method to move the arm head close to the target with quad-partitioning relationships in the camera’s visual space by using image data processing techniques. The error distance after k iterations of the proposed method is approximately estimated as (Formula presented), where (Formula presented) and (Formula presented) are the width and the length of the field of view. We conduct a case study to validate the proposed method. Testing results show that the proposed system successfully picks bean defects with our proposed robotic arm guidance method.