Model-based Building Reconstruction (MBBR) is a convinced approach to reconstructing 3D building models by fitting pre-defined models to aerial photographs. In this paper, we proposed the Floating Models, which can be applied to versatile data, such as the topographic maps, LiDAR points cloud, aerial photographs and DEM, with the ad hoc Least- Squares Model-Data Fitting (LSMDF) algorithm. Floating models is a series of pre-defined primitive models which are floating in the space as the floating mark is. Each model is associated with a set of parameters that can be used to control its shape and pose. A building is reconstructed by adjusting these parameters so the wire-frame model adequately fits into the building's outlines among versatile source data. This model-based reconstruction provides better constraints to the shape of the model in contrary to the data-based approach. Since topographic maps and aerial photographs provide better accuracy in plane and LiDAR points cloud and DEM provide better accuracy in height, the fitting are separated for plane and height parameters. The plane parameters are determined by fitting the top or bottom boundary of the model to the topographic maps. The height parameters are decided by fitting the top surface of the model to the lidar data and interpolating the datum point's height from DEM. The proposed reconstructing procedure is semi-automated. First, the operator chooses an appropriate model and approximately fit to the building's outlines on the topographic map. Second, the computer computes the optimal fit between the model and the building's boundaries based on the LSMDF algorithm. Third, the computer computes the roof or ridge height from the LiDAR points within the roof's boundary, and interpolates the datum height from DEM. Finally, the model parameters and standard deviations are provided and the wire-frame model is superimposed on all overlapped aerial photographs for manual checking. The operator can make any necessary modification by adjusting the corresponding model parameter. A 528-hectare urban area of Taipei City is selected for experimental tests. The corners' coordinates derived from fitting result are compared to the traditionally photogrammetric measurements. The experiment shows that most of the modern buildings can be modeled smoothly and the results achieve the accuracy as traditional Photogrammetry does.