Many consumer embedded system products nowadays are built on platforms with System-On-a-Chip (SOC) in which two or more processor cores, which are not necessarily of the same type, are put into a single chip and form the architecture of Chip-level Multi-Processor (CMP). Although such platform is capable of achieving high performance at relatively low cost, the system architecture of CMP brings new challenges in system development and increases complexity in developing embedded software especially at the level of kernel or operating system. This paper presents our experience and some preliminary results from building a multi-kernel embedded system with high performance Inter-Process Communication (IPC) mechanism for application software running on the platform of a newly developed multi-core SOC, namely PAC Duo SOC, which is the latest product from the PAC (short for Parallel Architecture Core) Project implemented at Industry Technology Research Institute (ITRI) in Taiwan. PAC Duo SOC is a heterogeneous multi-processor SOC composed of one ARM926 core serving as the general purpose processor (GPP) and two ITRI PAC DSP cores serving as the special purpose processors (SPP). We ported Linux operating system to run on the ARM926 processor and ported the real-time kernel Micro-C/OS-II to run on one PAC DSP core, leaving the other PAC DSP core with the option of running either another Micro-C/OS-II or a different kernel. To address the issues in IPC, a high performance message-passing mechanism is developed. Its design not only takes application-specific requirements into account but also takes advantages of hardware features.