Hierarchical prosody conversion using regression-based clustering for emotional speech synthesis

Chung Hsien Wu, Chi Chun Hsia, Chung Han Lee, Mai Chun Lin

Research output: Contribution to journalArticlepeer-review

43 Citations (Scopus)


This paper presents an approach to hierarchical prosody conversion for emotional speech synthesis. The pitch contour of the source speech is decomposed into a hierarchical prosodic structure consisting of sentence, prosodic word, and subsyllable levels. The pitch contour in the higher level is encoded by the discrete Legendre polynomial coefficients. The residual, the difference between the source pitch contour and the pitch contour decoded from the discrete Legendre polynomial coefficients, is then used for pitch modeling at the lower level. For prosody conversion, Gaussian mixture models (GMMs) are used for sentence- and prosodic word-level conversion. At subsyllable level, the pitch feature vectors are clustered via a proposed regression-based clustering method to generate the prosody conversion functions for selection. Linguistic and symbolic prosody features of the source speech are adopted to select the most suitable function using the classification and regression tree for prosody conversion. Three small-sized emotional parallel speech databases with happy, angry, and sad emotions, respectively, were designed and collected for training and evaluation. Objective and subjective evaluations were conducted and the comparison results to the GMM-based method for prosody conversion achieved an improved performance using the hierarchical prosodic structure and the proposed regression-based clustering method.

Original languageEnglish
Article number5289985
Pages (from-to)1394-1405
Number of pages12
JournalIEEE Transactions on Audio, Speech and Language Processing
Issue number6
Publication statusPublished - 2010

All Science Journal Classification (ASJC) codes

  • Acoustics and Ultrasonics
  • Electrical and Electronic Engineering


Dive into the research topics of 'Hierarchical prosody conversion using regression-based clustering for emotional speech synthesis'. Together they form a unique fingerprint.

Cite this