Various gesture labeling systems aim to establish guidelines on how to systematically assess and
annotate gestures. However, there is a lack of resources that provide guidance and training on the
proper use of the labeling systems. To address this gap, the present study aims to (a) collaboratively
develop an online training program for the M3D gesture labeling system (Rohrer et al., 2023) using
the ADDIE paradigm (Branch, 2009) and (b) to evaluate its usability, acceptability, feasibility and
effectiveness. ...
Various gesture labeling systems aim to establish guidelines on how to systematically assess and
annotate gestures. However, there is a lack of resources that provide guidance and training on the
proper use of the labeling systems. To address this gap, the present study aims to (a) collaboratively
develop an online training program for the M3D gesture labeling system (Rohrer et al., 2023) using
the ADDIE paradigm (Branch, 2009) and (b) to evaluate its usability, acceptability, feasibility and
effectiveness. Developing a training program for gesture annotation is essential for advancing the field
of gesture studies, equally important is conducting an evaluative process to ensure the program’s high
quality nature.
The evaluation will employ a mixed-methods approach, using the System Usability Scale (SUS)
(Brooke, 2013) for usability assessment, an ADDIE questionnaire and an in-house questionnaire for
acceptability, a starting-and-completion ratio for feasibility and an inter-annotator reliability
assessment for effectiveness. Thirteen participants took part in the evaluation. The results show that
the program has achieved a remarkable degree of usability, especially in terms of ease of use,
complexity and integration of functionalities, reaching an overall score of “Excellent”, with individual
scores ranging from 70-97.7 on a 1-100 scale. In terms of acceptability, it was found to enhance the
participants’ skills (M= 4.90, SD= 0.316), learning material was perceived as supporting the learning
process (M= 4.90, SD= 0.316) and value was added by visual material (M= 5.00, SD= 0.00). The
starting-and-completion ratio stands at 13:10. Effectiveness shows high inter-annotator reliability
results for Gesture Phases and Gesture Referentiality. Overall, the projects’ results represent a major
advancement for gesture studies, offering for the first time an online gesture annotation training
program which moves the field towards a more standardized approach to assessing gesture in
multimodal corpora.
+
