M3D Gesture Labeling: developing and evaluating a training program for gesture annotation - a mixed-methods study

Abstract

Various gesture labeling systems aim to establish guidelines on how to systematically assess and annotate gestures. However, there is a lack of resources that provide guidance and training on the proper use of the labeling systems. To address this gap, the present study aims to (a) collaboratively develop an online training program for the M3D gesture labeling system (Rohrer et al., 2023) using the ADDIE paradigm (Branch, 2009) and (b) to evaluate its usability, acceptability, feasibility and effectiveness. Developing a training program for gesture annotation is essential for advancing the field of gesture studies, equally important is conducting an evaluative process to ensure the program’s high quality nature. The evaluation will employ a mixed-methods approach, using the System Usability Scale (SUS) (Brooke, 2013) for usability assessment, an ADDIE questionnaire and an in-house questionnaire for acceptability, a starting-and-completion ratio for feasibility and an inter-annotator reliability assessment for effectiveness. Thirteen participants took part in the evaluation. The results show that the program has achieved a remarkable degree of usability, especially in terms of ease of use, complexity and integration of functionalities, reaching an overall score of “Excellent”, with individual scores ranging from 70-97.7 on a 1-100 scale. In terms of acceptability, it was found to enhance the participants’ skills (M= 4.90, SD= 0.316), learning material was perceived as supporting the learning process (M= 4.90, SD= 0.316) and value was added by visual material (M= 5.00, SD= 0.00). The starting-and-completion ratio stands at 13:10. Effectiveness shows high inter-annotator reliability results for Gesture Phases and Gesture Referentiality. Overall, the projects’ results represent a major advancement for gesture studies, offering for the first time an online gesture annotation training program which moves the field towards a more standardized approach to assessing gesture in multimodal corpora.