In this paper, we outline an approach of end-to-end interactive systems with emotional embeddings, which are transfered from a large corpus. We show how to apply emotional embeddings trained from Twitter databases with hashtags and emojis as labels in a regression task. We also show that task-oriented dialog systems can be cast in an end-to-end framework using recurrent entity networks and dynamic query memory networks. In addition, we propose to include emotional embeddings into this framework for a more empathetic human-machine interactions. Finally, we show how to train an end-to-end open-domain dialog systems with deep reinforcement learning that learns a sense of humour from TV sitcoms.