Generative pre-trained transformers for coding text data? An analysis with classroom orchestration data

Mostra el registre complet Registre parcial de l'ítem

• dc.contributor.author Amarasinghe, Ishari
• dc.contributor.author Marques, Francielle
• dc.contributor.author Ortiz-Beltran, Ariel
• dc.contributor.author Hernández Leo, Davinia
• dc.date.accessioned 2023-06-27T07:04:10Z
• dc.date.issued 2023
• dc.description Comunicació presentada a la 18th European Conference on Technology Enhanced Learning (EC-TEL 2023), celebrada a Aveiro (Portugal) del 4 al 8 de setembre de 2023.
• dc.description.abstract Video content analysis is of importance for researchers in technology-enhanced learning. A common starting point typically involves transcribing video into textual transcripts that enable the application of a coding scheme to group the text into key themes. However, manual coding is demanding and requires time and effort of human annotators. Therefore, this study explores the possibility of using Generative Pre-trained Transformer 3 (GPT-3) models for automating the text data coding compared to baseline classical machine learning approaches using a dataset manually coded for the orchestration actions of six teachers in classroom collaborative learning sessions. The findings of our study showed that a fine-tuned GPT-3 (curie) model outperformed classical approaches (F1 score of 0.87) and reached a 0.77 Cohen’s kappa, which indicated a moderate agreement between manual and machine coding. The study also brings out the limitations of our text transcripts and highlights the importance of multimodal observations that capture the context of orchestration actions.
• dc.description.sponsorship This work has been partially funded by AEI/10.13039/501100011033 (PID2020-112584RB-C33) and (PLAWB00322). DHL (Serra Hunter) also acknowledges the support by ICREA under the ICREA Academia programme.
• dc.format.mimetype application/pdf
• dc.identifier.citation Amarasinghe I, Marques F, Ortiz-Beltran A, Hernández-Leo D. Generative pre-trained transformers for coding text data? An analysis with classroom orchestration data. Paper presented at: 18th European Conference on Technology Enhanced Learning (EC-TEL 2023); 2023 Sep 4-8; Aveiro, Portugal.
• dc.identifier.uri http://hdl.handle.net/10230/57376
• dc.language.iso eng
• dc.publisher Springer
• dc.relation.projectID info:eu-repo/grantAgreement/ES/2PE/PID2020-112584RB-C33
• dc.rights © Springer. This is a author's accepted manuscript of: Amarasinghe I, Marques F, Ortiz-Beltran A, Hernández-Leo D. Generative pre-trained transformers for coding text data? An analysis with classroom orchestration data. Paper presented at: 18th European Conference on Technology Enhanced Learning (EC-TEL 2023); 2023 Sep 4-8; Aveiro, Portugal.
• dc.rights.accessRights info:eu-repo/semantics/openAccess
• dc.subject.keyword Collaborative learning
• dc.subject.keyword Automatic coding
• dc.subject.keyword Machine Learning
• dc.subject.keyword Generative AI
• dc.subject.keyword Classroom orchestration
• dc.title Generative pre-trained transformers for coding text data? An analysis with classroom orchestration data
• dc.type info:eu-repo/semantics/conferenceObject
• dc.type.version info:eu-repo/semantics/acceptedVersion