This paper presents a scheme for the reconstruction of building models from lidar data and 2D maps by using a split and merge approach. The proposed scheme comprises of three major parts: (1) data preprocessing (2) extraction of building primitive, and (3) building shaping. In the preprocessing, a grid digital surface model and triangulated irregular networks are both generated from the lidar point clouds. The building boundaries are traced to form closed polygons. In the extraction of building primitive, we extract the inner structure lines from lidar data. Then, we split the building polygon into several building primitive by the extracted feature lines. In the building reconstruction, we shape the roof for each building primitive by using lidar point clouds. Finally, we merge all the 3D building primitives to obtain the building models. The proposed method has been tested with the data at Ping-Dong city of south Taiwan. Experimental results indicate that the proposed scheme reaches high fidelity.