Content validity of tests applied in an oral comprehension course for academic purposes: an experience report

Abstract

This article reports the analysis of the content validity (GREEN 2014) of tests applied in an English for Academic Purposes course focusing on oral comprehension taught at the Languages without Borders Program. The aim was to verify the alignment between the teaching plan and the tests. The categories for analysis were the micro and macroskills according to Brown (2004; 2007). Questionnaires were used to collect students’ perceptions about the questions’ clarity and the audio’s level of difficulty. The results indicated that the first test presented satisfactory content valididy, two not very difficult audios and questions that did not pose comprehension problems. In the second test, content validity was partial (60%) and judgment concerning question clarity and audio difficulty were the same as before. As for the third test, it was considered content valid, though it was complex to verify that because there was no explicit correspondence between the teaching plan and the question rubrics. In relation to the clarity of the questions, it was the same as for the previous tests, and the audio was the one students considered the most difficult, leading us to reflections on possible reasons for that.