Zara: A Virtual Interactive Dialogue System Incorporating Emotion, Sentiment and Personality Recognition

Published in The Conference on Computational Linguistics (COLING Demo), 2016

[PDF]


@inproceedings{fung2016zara,
  title={Zara: A Virtual Interactive Dialogue System Incorporating Emotion, Sentiment and Personality Recognition},
  author={Fung, Pascale and Dey, Anik and Siddique, Farhad Bin and Lin, Ruixi and Yang, Yang and Bertero, Dario and Wan, Yan and Chan, Ricky Ho Yin and Wu, Chien-Sheng},
  booktitle={Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: System Demonstrations},
  pages={278--281},
  year={2016}
}

Abstract

Zara, or ‘Zara the Supergirl’ is a virtual robot, that can exhibit empathy while interacting with an user, with the aid of its built in facial and emotion recognition, sentiment analysis, and speech module. At the end of the 5-10 minute conversation, Zara can give a personality analysis of the user based on all the user utterances. We have also implemented a real-time emotion recognition, using a CNN model that detects emotion from raw audio without feature extraction, and have achieved an average of 65.7% accuracy on six different emotion classes, which is an impressive 4.5% improvement from the conventional feature based SVM classification. Also, we have described a CNN based sentiment analysis module trained using out-of-domain data, that recognizes sentiment from the speech recognition transcript, which has a 74.8 F-measure when tested on human-machine dialogues.