Previous studies have found systematic associations between personality and individual differences in interpersonal communication. Recently, researchers used various features to analyze individual personality traits in speech, social media content and essays. While few studies focused on detecting the personality interaction between two interlocutors, this paper presents a new approach to automatically and simultaneously predict the personalities of two interlocutors in a dyadic conversation. First, the recurrent neural networks (RNNs) are adopted to project the linguistic features of the transcribed spoken text of the input speech to the Big Five Inventory (BFI) space. The Coupled hidden Markov models (coupled HMMs) are then used to predict the interlocutor personality from the transcribed text of the two speakers considering the conversational interaction in their dialogue turns. The Mandarin Conversational Dialogue Corpus (MCDC) was adopted to evaluate the performance on interlocutor personality perception. Experimental results show that the proposed approach achieved satisfactory results in predicting personalities of two interlocutors at the same time.