Assessing complexity and difficulty levels of machine-translated texts

Abstract

This paper addresses a proposal for assessing complexity and difficulty levels of machine-translated texts in Portuguese to be further post-edited without the support of the source text (monolingual post-editing) in an experimental setting. By using two objective standard parameters, namely readability indexes and word frequency, and by proposing post-editors' perception of difficulty to comprehend and to post-edit machine-translated texts as a new parameter, we sought to select texts with similar textual complexity or difficulty levels. This selection was necessary to carry out an experiment with four monolingual post-editing tasks in Portuguese involving machine-translated texts from three different source languages (English, Spanish, and Chinese). The application of readability indexes in conjunction with word frequency based on a corpus to analyze machine-translated texts into Portuguese to be used in experiments showed to be consistent and adequate. This method can also be applied to select texts to be used in Portuguese language classrooms and to select Portuguese texts to be included in Portuguese language textbooks. The findings can also be applied to the translation classroom, in which teachers can use the same methodology to select texts to be translated or post-edited or encourage students to analyze the texts themselves before performing a task, so students can become aware of the potential effort to be invested on a task or the real effort invested on the task after performing it. Finally, post-editors' perception proved to be a sound parameter to validate text selection.