Convolutional neural network language models

Mostra el registre complet Registre parcial de l'ítem

• dc.contributor.author Boleda, Gemmaca
• dc.contributor.author Pham, Nghia Theca
• dc.contributor.author Kruszewski, Germanca
• dc.date.accessioned 2017-08-25T17:17:01Z
• dc.date.available 2017-08-25T17:17:01Z
• dc.date.issued 2016
• dc.description.abstract Convolutional Neural Networks (CNNs) have shown to yield very strong results in several Computer Vision tasks. Their application to language has received much less attention, and it has mainly focused on static classification tasks, such as sentence classification for Sentiment Analysis or relation extraction. In this work, we study the application of CNNs to language modeling, a dynamic, sequential prediction task that needs models to capture local as well as long-range dependency information. Our contribution is twofold. First, we show that CNNs achieve 11-26% better absolute performance than feed-forward neural language models, demonstrating their potential for language representation even in sequential tasks. As for recurrent models, our model outperforms RNNs but is below state of the art LSTM models. Second, we gain some understanding of the behavior of the model, showing that CNNs in language act as feature detectors at a high level of abstraction, like in Computer Vision, and that the model can profitably use information from as far as 16 words before the target.en
• dc.description.sponsorship This project has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No 655577 (LOVe); ERC 2011 Starting Independent Research Grant n. 283554 (COMPOSES) and the Erasmus Mundus Scholarship for Joint Master Programs.en
• dc.format.mimetype application/pdf
• dc.identifier.citation Pham N, Kruszewski G, Boleda G. Convolutional neural network language models. In: Su J, Duh K, Carreras X, editors. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Austin: Association for Computational Linguistics; 2016. p. 1153-1162.
• dc.identifier.uri http://hdl.handle.net/10230/32701
• dc.language.iso eng
• dc.publisher ACL (Association for Computational Linguistics)ca
• dc.relation.ispartof Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Austin: Association for Computational Linguistics; 2016. p. 1153-1162
• dc.relation.projectID info:eu-repo/grantAgreement/EC/H2020/655577
• dc.relation.projectID info:eu-repo/grantAgreement/EC/FP7/283554
• dc.rights © ACL, Creative Commons Attribution 4.0 License
• dc.rights.accessRights info:eu-repo/semantics/openAccess
• dc.rights.uri http://creativecommons.org/licenses/by/4.0/
• dc.subject.keyword Convolutional Neural Networks
• dc.subject.keyword CNNs
• dc.subject.keyword Natural Language Processing
• dc.subject.keyword Deep learningen
• dc.subject.keyword Language modelsen
• dc.title Convolutional neural network language modelsca
• dc.type info:eu-repo/semantics/conferenceObject
• dc.type.version info:eu-repo/semantics/publishedVersion