Study and correlation analysis of linguistic, perceptual, and automatic machine translation evaluations

Abstract

Evaluation of machine translation output is an important task. Various human evaluation techniques as well as automatic metrics have been proposed and investigated in the last decade. However, very few evaluation methods take linguistic aspect into account. In this paper, we use an objective evaluation method for machine translation output that classifies all translation errors into one of the five following linguistic levels: orthographic, morphological, lexical, semantic and syntactic, in order to analyse its linguistic quality. Linguistic guidelines for the target language are required, and human evaluators use them in to classify the output errors. The experiments are performed on English-to-Catalan and Spanish-to-Catalan translation outputs generated by four different systems: two rule-based and two statistical. All translations are evaluated using three following methods: a standard human evaluation method, several widely used automatic metrics, and the linguistic evaluation. Pearson and Spearman correlation coefficients between the linguistic, human and automatic results are then calculated, showing that the semantic level correlates significantly with both human evaluation and automatic metrics.