Cases of physical and mental diseases caused by stress and negative emotions have increased annually. Many emotion recognition methods have been proposed. Facial expression is widely used for emotion recognition. However, since facial expressions may be expressed differently by different people, inaccurate results are unavoidable. Nerve and Physiological responses are incontrollable native response. Physiological responses and the corresponding signals are difficult to control when a person is overcome with emotion. Therefore, an emotion recognition system that considers physiological signals is proposed in this paper. An emotion induction experiment was performed to collect five physiological signals from subjects, namely electrocardiogram, respiration, galvanic skin response (GSR), blood volume pulse, and pulse. Support vector regression (SVR) was used to train three trend curves of three emotions (sadness, fear, and pleasure). Experimental results show that the proposed method has a high recognition rate of 90.6%.