Three-dimensional integrated circuits are expected to face increasingly severe thermal challenges and cost issues as the number of stacked ICs increases. Thermal analysis for 3D ICs is urgently required to assist system designers at the early phase of design to identify hot zones. Most thermal analyses obtain detailed temperature distribution by large matrix operations, and hence reduce analysis performance. Accordingly, we propose a compressed and combined sparse row (CCSR) matrix format to be used in the proposed effective matrix compression (EMC) method for matrix multiplication on GPU. The experimental results show EMC using CCSR is on average 44.93 times faster than matrix multiplication without any special compression format and on average at least 3.09 times faster than other compression formats.