Studies on color quantization have indicated that its applications range from the relaxation of displaying hardware constraints in early years to a modern usage of facilitating content-based image retrieval tasks. Among many alternatives, approaches based on neural network models are generally accepted to be very effective in color quantization. However, the inefficiency prevents their usefulness from practical usage. In view of this, we thus propose to incorporate a growing quadtree structure to the self-organizing map (GQSOM) technique in this work. Specifically, the strategy of inheriting from parent neurons hierarchically facilitates a much more efficient and flexible learning process. Both theoretical and empirical studies show that our approach is adaptive in determining an appropriate number of quantized colors. Moreover, the efficiency is significantly improved without compromise of the effectiveness.