Coffee beans are one of most valuable agricultural products in the world, and defective bean removal plays a critical role to produce high-quality coffee products. In this work, we propose a novel labor-efficient deep learning-based model generation scheme, aiming at providing an effective model with less human labeling effort. The key idea is to iteratively generate new training images containing defective beans in various locations by using a generative-adversarial network framework, and these images incur low successful detection rate so that they are useful for improving model quality. Our proposed scheme brings two main impacts to the intelligent agriculture. First, our proposed scheme is the first work to reduce human labeling effort among solutions of vision-based defective bean removal. Second, our scheme can inspect all classes of defective beans categorized by the SCAA (Specialty Coffee Association of America) at the same time. The above two advantages increase the degree of automation to the coffee industry. We implement the prototype of the proposed scheme for conducting integrated tests. Testin. results of a case study reveal that the proposed scheme ca] efficiently and effectively generating models for identifyin defect beans.Our implementation of the proposed scheme is available a https://github.com/Louis8582/LEGAN.