Multimodal user's affective state analysis in naturalistic interaction

George Caridakis, Kostas Karpouzis, Manolis Wallace, Loic Kessous, Noam Amir

Research output: Contribution to journalArticlepeer-review


Affective and human-centered computing have attracted an abundance of attention during the past years, mainly due to the abundance of environments and applications able to exploit and adapt to multimodal input from the users. The combination of facial expressions with prosody information allows us to capture the users' emotional state in an unintrusive manner, relying on the best performing modality in cases where one modality suffers from noise or bad sensing conditions. In this paper, we describe a multi-cue, dynamic approach to detect emotion in naturalistic video sequences, where input is taken from nearly real world situations, contrary to controlled recording conditions of audiovisual material. Recognition is performed via a recurrent neural network, whose short term memory and approximation capabilities cater for modeling dynamic events in facial and prosodic expressivity. This approach also differs from existing work in that it models user expressivity using a dimensional representation, instead of detecting discrete 'universal emotions', which are scarce in everyday human-machine interaction. The algorithm is deployed on an audiovisual database which was recorded simulating human-human discourse and, therefore, contains less extreme expressivity and subtle variations of a number of emotion labels. Results show that in turns lasting more than a few frames, recognition rates rise to 98%.

Original languageEnglish
Pages (from-to)49-66
Number of pages18
JournalJournal on Multimodal User Interfaces
Issue number1
StatePublished - Mar 2010
Externally publishedYes


  • Affective computing
  • Emotion dynamics
  • Emotion recognition
  • Multimodal analysis
  • Recurrent neural network


Dive into the research topics of 'Multimodal user's affective state analysis in naturalistic interaction'. Together they form a unique fingerprint.

Cite this