Farrús, MireiaCosta-jussà, Marta R.Popovic, Maja2018-04-302018-04-302011Farrús M, Costa-Jussà MR, Popovic M. Study and correlation analysis of linguistic, perceptual, and automatic machine translation evaluations. J Am Soc Inf Sci. 2012;63(1):174-84. DOI: 10.1002/asi.216742330-1635http://hdl.handle.net/10230/34518Evaluation of machine translation output is an important task. Various human evaluation techniques as well as automatic metrics have been proposed and investigated in the last decade. However, very few evaluation methods take linguistic aspect into account. In this paper, we use an objective evaluation method for machine translation output that classifies all translation errors into one of the five following linguistic levels: orthographic, morphological, lexical, semantic and syntactic, in order to analyse its linguistic quality. Linguistic guidelines for the target language are required, and human evaluators use them in to classify the output errors. The experiments are performed on English-to-Catalan and Spanish-to-Catalan translation outputs generated by four different systems: two rule-based and two statistical. All translations are evaluated using three following methods: a standard human evaluation method, several widely used automatic metrics, and the linguistic evaluation. Pearson and Spearman correlation coefficients between the linguistic, human and automatic results are then calculated, showing that the semantic level correlates significantly with both human evaluation and automatic metrics.application/pdfengThis is the peer reviewed version of the following article: "Farrús M, Costa-Jussà MR, Popovic M. Study and correlation analysis of linguistic, perceptual, and automatic machine translation evaluations. J Am Soc Inf Sci. 2012; 63 (1): 174-84.", which has been published in final form at http://dx.doi.org/10.1002/asi.21674. This article may be used for non-commercial purposes in accordance with Wiley Terms and Conditions for Self-Archiving.info:eu-repo/semantics/articlehttp://dx.doi.org/10.1002/asi.21674Machine translationLinguistic evaluationAutomatic evaluationHuman evaluationCatalaninfo:eu-repo/semantics/openAccess