LIDAR data contains plenty of height information, while vector maps preserve accurate building boundaries. From the viewpoint of data fusion, we integrate LIDAR data and large-scale vector maps to perform building modeling. The proposed scheme comprises six major steps: (1) preprocessing of LIDAR data and vector maps, (2) extraction of point clouds that belong to a building, (3) construction of the facets from the point clouds, (4) detection of planar faces, (5) determination of 3-D edges of buildings, and (6) regularization of 3-D edges and building reconstruction. In the preprocessing stage, the height variation of the aboveground objects is extracted by subtracting the surface elevation from the terrain. The polygons for buildings are also obtained from the polylines using the SMS method. Using the vertex locations and rough heights of stories, the point clouds that belong to a building can be selected. Then a triangulated irregular network is built for representing the facets of the point clouds. Segmentation of planar faces is implemented by examining the size and the angles among surface normal vectors. After detection for planar roof faces, 3-D roof edges are determined by intersecting roof planes. Finally, building models are reconstructed after regularization.