A hybrid recommendation system considering visual information for predicting favorite restaurants

Restaurant recommendation is one of the most interesting recommendation problems because of its high practicality and rich context. Many works have been proposed to recommend restaurants by considering user preference, restaurant attributes, and socio-demographic behaviors. In addition to these, many customers review restaurants in blog articles where text-based subjective comments and various photos may be available. In this paper, we especially investigate the influence of visual information, i.e., photos taken by customers and put on blogs, on predicting favorite restaurants for any given user. By considering visual information as the intermediate, we will integrate two common recommendation approaches, i.e., content-based filtering and collaborative filtering, and show the effectiveness of considering visual information. More particularly, we advocate that, in addition to text information or metadata, restaurant attributes and user preference can both be represented by visual features. Heterogeneous items can thus be modeled in the same space, and thus two types of recommendation approaches can be linked. Through experiments with various settings, we verify that visual information effectively aids favorite restaurant prediction.