Computational modeling of user activity in full-body interaction environments for ASC children: multimodal analysis of social interaction behaviors through psychophysiology, system activity, video coding, questionnaires, and body cues

Sayis, Batuhan

Computational modeling of user activity in full-body interaction environments for ASC children: multimodal analysis of social interaction behaviors through psychophysiology, system activity, video coding, questionnaires, and body cues

Document Type

Doctoral thesis
Published Version

Author

Sayis, Batuhan

Director

Parés, Narcís
Ramírez, Rafael

Publication Date

2021-05-04

Pages

154 p.

License

Citation

Sayis, B. Computational modeling of user activity in full-body interaction environments for ASC children: multimodal analysis of social interaction behaviors through psychophysiology, system activity, video coding, questionnaires, and body cues. Universitat Pompeu Fabra; 2021. handle: http://hdl.handle.net/10803/671850

This citation was generated automatically.

Doctoral program

Programa de doctorat en Tecnologies de la Informació i les Comunicacions

Abstract

Full-body Interaction experiences based on Mixed Reality (MR) systems are already playing an important role in encouraging socialization behaviors in children with Autism Spectrum Condition (ASC), as seen in the state of the art of this thesis. However, the data from these systems is multimodal in nature and complex to analyze. Fusion and analysis of this data is crucial to achieve a complete understanding of how these resources interact with each other. In this PhD Thesis, given the characteristics of full-body interaction, we developed new multimodal data gathering and evaluation techniques to better understand the effectiveness of the experience developed in our Full-body Interaction Lab (FuBIntLab) called Lands of Fog. This is a large-scale MR, full-body interaction environment, which allows two children to play face-to-face and explore the physical and virtual worlds simultaneously. Specifically, we developed an experimental setup for comparing Lands of Fog with a control condition based on LEGO construction toys, which includes: recording psychophysiological measures synchronized with other data sources such as observed overt behaviors and system logs of game events. In order to capture accurate psychophysiological data, we developed a wearable that is child-friendly and robust to movement artifacts in the context of ambulatory full-body interaction. In order to integrate observed overt behaviors with other data sources, we designed and developed a novel video coding protocol and an adapted coding grid conceived for Social Interaction Behaviors (SIBs) in ASC children. Using a repeated measure design, we collected data from seventy-two children (36 ASC/non-ASC dyads) from the city of Barcelona, with ages between 8-12 years old (N = 12 female, N = 60 male). Data from these trials has been organized into a public database and processed based on a semi-automatic software pipeline developed within this project. Based on this data we developed three different computational models for modelling SIBs in children with ASC during Lands of Fog sessions, compared to LEGO sessions. The results of this research support the idea that full-body interaction MR environments are capable of fostering SIBs in children with ASC with similar success as the LEGO setting, with an added advantage of being more flexible. Findings reported here shed new light on developing a tool that is mediating, guiding, and supporting the progress of the children in terms of practicing SIBs and providing structure and assistance to therapists.

Keywords

Machine learning, Multimodal evaluation, Physiological measures, Wearables, Mixed reality, Embodied interaction, Autism spectrum condition, Children, Social interaction, Aprendizaje automático, Evaluación multimodal, Medidas fisiológicas, Realidad mixta, Interacción corpórea, Trastorno del espectro autista, Niños, Interacción social, Aprenentatge automàtic, Avaluació multimodal, Mesures fisiològiques, Realitat mixta, Interacció corpòria, Trastorn de l'espectre autista, Infants, Interacció social