‘Seeing tunes.’ The role of visual gestures in tune interpretation

Borràs Comes, Joan Manel, 1984-; Prieto Vives, Pilar, 1965-

‘Seeing tunes.’ The role of visual gestures in tune interpretation

Mostra el registre complet Registre parcial de l'ítem

dc.contributor.author Borràs Comes, Joan Manel, 1984-ca
dc.contributor.author Prieto Vives, Pilar, 1965-ca
dc.date.accessioned 2016-12-20T15:06:41Z
dc.date.available 2016-12-20T15:06:41Z
dc.date.issued 2011
dc.description.abstract One of the unresolved questions in audiovisual prosody is the relative contribution of acoustic and visual cues to the expression of prosodic meaning. Though the majority of studies on audiovisual prosody have found a complementary mode of processing whereby sight provides relatively weak and redundant information in comparison with strong auditory cues, other work has found that sight provides information more efficiently than hearing. In Catalan, a pitch range contrast in a rising-falling nuclear configuration conveys a difference between a contrastive focus statement and an echo question. The main goal of this study is to investigate the relative contribution of visual cues in conveying this distinction. Twenty native speakers of Central Catalan participated in two identification tasks in which they had to decide between a focus statement and a question interpretation. Experiment 1 used a pitch range auditory continuum combined with two congruent and incongruent videotapes showing the facial gestures that are characteristic of the two pragmatic meanings. Experiment 2 used the same auditory continuum in combination with another continuum for facial gestures produced using a digital imagemorphing technique. The responses and reaction times obtained in both experiments revealed a consistent reliance on visual cues in the listener’s decisions, but also a consistent effect of the auditory stimulus. We argue that although facial gestures are the most influential elements that Catalan listeners rely on to decide between contrastive focus and echo question interpretations, bimodal integration with the acoustic cues is necessary for perceptual processing to be accurate and fast. Finally, we discuss the implications of these results for models of audiovisual processing.en
dc.description.sponsorship We are grateful to Carme de la Mota, Itziar Laka, Lluís Payrató, Josep Quer, Núria Sebastián-Gallés, Maria-Josep Solé, Marc Swerts and Eric Vatikiotis-Bateson for their comments on an earlier version of this paper. We would like to thank the members of the Group of Prosodic Studies (UPF-UAB, Barcelona) for their help in managing E-Prime scripts and statistics. We would also like to thank the subjects who took part in the two experiments and the recordings of the audiovisual materials, especially Eva Estebas-Vilaplana, for their comments on the gestural materials. We also thank the editors of this special issue, Ian Maddieson and Caroline Smith, as well as Beatriz Raposo and two anonymous referees for their insightful comments, which helped clarify and strengthen this paper. This research has been funded by projects FFI2009-07648/ FILO and CONSOLIDER-INGENIO 2010 Programme CSD2007-00012 ( both awarded by the Ministerio de Ciencia e Innovación) and by project 2009 SGR 701 (awarded by the Generalitat de Catalunya).en
dc.format.mimetype application/pdfca
dc.identifier.citation Borràs-Comes J, Prieto P. 'Seeing tunes.' The role of visual gestures in tune interpretation. Lab Phonol. 2011 Oct 13; 2(2): 355-80. DOI: 10.1515/labphon.2011.013.en
dc.identifier.doi http://dx.doi.org/10.1515/labphon.2011.013
dc.identifier.issn 1868-6346
dc.identifier.uri http://hdl.handle.net/10230/27803
dc.language.iso engen
dc.publisher De Gruyteren
dc.relation.ispartof Journal for the Association of Laboratory Phonology. 2011 Oct 13; 2(2): 355-380en
dc.relation.projectID Info:eu-repo/grantAgreement/ES/3PN/FFI2009-07648
dc.relation.projectID Info:eu-repo/grantAgreement/Es/2PN/CSD2007-00012
dc.rights.accessRights info:eu-repo/semantics/openAccess
dc.subject.keyword Audiovisual prosodyen
dc.subject.keyword Facial gesturesen
dc.subject.keyword Pitch accent rangeen
dc.subject.keyword Intonational contrastsen
dc.subject.keyword Catalan languageen
dc.title ‘Seeing tunes.’ The role of visual gestures in tune interpretationen
dc.type info:eu-repo/semantics/article
dc.type.version info:eu-repo/semantics/publishedVersion

Col·leccions

Articles (Departament de Traducció i Ciències del Llenguatge)