We introduce a novel template-based modeling technique for 3D point clouds sampled from unknown buildings. The approach is based on a hierarchy algebraic template to fit point clouds with a significant amount of noises and sharp features. In the hierarchical template, the first-level, i.e., the lowest-level, contains three kinds of primitive geometries: plane, sphere, and cylinder. These primitive geometries are represented in algebra form. In the higher levels, some simple shapes are constructed by combining these primitive geometries, and the simple shapes can be further joined to form the final template model. In the fitting process, different to the general approaches which are intrinsic an iterative fitting process, we fit point clouds by directly solving a least-square linear system. This makes our approach efficient and robust in the modeling. Furthermore, some geometric constraints are integrated in the fitting process for the purpose of increasing modeling accuracy. The experiment results show that the modeling quality is improved, and the proposed template-based fitting is robust, in terms of withstanding noises and preserving sharp features, than the approaches based on implicit surfaces.