Sukno, Federico MateoDomínguez Bajo, MónicaRuiz, AdriàSchiller, DominikLingenfelser, FlorianPragst, LouisaKamateri, EleniVrochidis, Stefanos2016-07-262016-07-262016Sukno FM, Domínguez M, Ruiz A, Schiller D, Lingenfelser F, Pragst L, Kamateri E, Vrochidis S. A multimodal annotation schema for non-verbal affective analysis in the health-care domain. In: Proceedings of the 1st International Workshop on Multimedia Analysis and Retrieval for Multimodal Interaction (MARMI 2016); 2016 Jun 6; New York, USA. New York: ACM, 2016. p. 9-14. DOI: 10.1145/2927006.2927008http://hdl.handle.net/10230/27207The development of conversational agents with human interaction capabilities requires advanced affective state recognition integrating non-verbal cues from the different modalities constituting what in human communication we perceive as an overall affective state. Each of the modalities is often handled by a different subsystem that conveys only a partial interpretation of the whole and, as such, is evaluated only in/nterms of its partial view. To tackle this shortcoming, we investigate the generation of a unified multimodal annotation schema of non-verbal cues from the perspective of an interdisciplinary group of experts. We aim at obtaining a common ground-truth with a unique representation using the Valence and Arousal space and a discrete non-linear scale of values. The proposed annotation schema is demonstrated on/na corpus in the health-care domain but is scalable to other purposes. Preliminary results on inter-rater variability show a positive correlation of consensus level with high (absolute) values of Valence and Arousal as well as with the number of annotators labeling a given video sequence.application/pdfeng© ACM, 2016. This is the author's version of the work. It is posted here by permission of ACM for your personal use. Not for redistribution. The definitive version was published in Proceedings of the 1st International Workshop on Multimedia Analysis and Retrieval for Multimodal Interaction (MIRMI 2016). http://doi.acm.org/10.1145/2927006.2927008info:eu-repo/semantics/conferenceObjecthttp://dx.doi.org/10.1145/2927006.2927008Valence-arousalHuman-machine interactionMultimodal analysisEmbodied conversational agentsinfo:eu-repo/semantics/openAccess