Online location-based services, such as Foursquare and Facebook, provide a great resource for location recommendation. As we know the time is one of the important factors on recommending places with proper time for users, since the pleasure of visiting a place could be diminished if arriving at wrong time, we propose to infer the visiting time distributions of locations. We assume the check-in data used is incomplete because in real-world scenarios it is hard or unavailable to collect all the temporal information of locations and the check-in behaviors might be abnormal. To tackle such problem, we devise a visiting time inference framework, VisTime-Miner, which considers the route-based visiting correlation of time labels to model the visiting behavior of a location. Experiments on a large-scaled Gowalla check-in data show a promising result.