Many image search engines nowadays still struggle with the semantic gap between low level image features and high level image concepts. Some solutions are proposed to bridge the gap by using surrounding texts of images or by adding tags on images by single user. However, they can only provide obscure or limited information about images. Another problem is that users may not know exactly what they want when they search for images. In this work, we proposed a Progressive Image Search And Recommendation system, named as PISAR, to reduce the semantic gap by incorporating the auto-interpretation and user behavior. PISAR is able to progressively improve the interpretation of images and provide a list of recommendation. The evaluation results show that with the help of auto-interpretation and user behavior, the performance of search results and recommendation results can be progressively improved.