Generative AI chatbot in PyramidApp: students’ behaviors and design principles

Abstract

Generative Artificial Intelligence (GenAI) offers new opportunities to implement useful features within Computer Supported Collaborative Learning (CSCL) environments. Despite these growing prospects, there is still limited research concerning the application of GenAI in learning environments. This work in progress aims to evaluate the mediation of a masked GenAI chatbot in the setting of the CSCL web application PyramidApp. A quasi-experimental within-subjects study was designed to assess the effect of GenAI chatbot intervention within the environment of PyramidApp. In the setting of 9 online activities, we evaluated the effect of the GenAI chatbot activity in 105 conversational chat rooms. The findings revealed that the GenAI chatbot provides useful feedback as students rate the chatbot’s answers higher than their peers’ answers (MChatbot = 4.11, MStudents = 3.91). The presence of the chatbot has an effect on group communication with the length of messages increased in chat rooms where the chatbot was present. Moreover, chatbot behavior to rate the students’ answers was correlated with the students’ behavior. The present study offers valuable insights into the optimal strategies for integrating a GenAI Large Language Model into educational tools and computer supported learning.