ITEM RESPONSE THEORY ANALYSIS OF THE READING SECTION IN A MOCK VSTEP LEVEL 3-5 TEST
DOI:
https://doi.org/10.18173/2354-1075.2025-0064Keywords:
Item Response Theory, VSTEP, Reading, reading proficiencyAbstract
This study investigates the use of Item Response Theory (IRT) to evaluate and validate the items in the reading section of a mock Vietnamese Standardized Test of English Proficiency (VSTEP) level 3-5 according to the Vietnamese version of the Common European Framework of Reference (CEFR). The primary focus is to evaluate how well the test items align with the abilities of test-takers, comparing the question difficulty as designed (R-level) with the estimated difficulty generated through IRT analysis. Data were collected from responses of 100 test-takers to the Reading section of the mock VSTEP Level 3-5. By applying IRT, the study offers a detailed analysis of item characteristics, including item difficulty and discrimination parameters, which are crucial for validating whether the test accurately reflects the proficiency levels it is intended to measure. The results provide insights into the validity and reliability of the test items, highlighting potential discrepancies between the original design of question difficulty and the actual performance data. This analysis contributes to the overall improvement of the test, ensuring that it better serves as an accurate tool for assessing language proficiency at the specified levels. The findings also offer valuable implications for future test development and refinement in similar contexts.
Downloads
References
[1] Kane, M. T. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50(1), 1–73. https://doi.org/10.1111/jedm.12000
[2] Brunfaut, T. (2023). Future challenges and opportunities in language testing and assessment: Basic questions and principles at the forefront. Language Testing, 40(1), 15–23. https://doi.org/10.1177/02655322221127896
[3] An, X., & Yung, Y. F. (2014). Item response theory: What it is and how you can use the IRT procedure to apply it. SAS Institute.
[4] Min, S., & Aryadoust, V. (2021). A systematic review of item response theory in language assessment: Implications for the dimensionality of language ability. Studies in Educational Evaluation, 68, 100963. https://doi.org/10.1016/j.stueduc.2021.100963
[5] Crocker, L., & Algia, J. (1986). Introduction to classical & modern test theory. Wadsworth.
[6] Henard, D. H. (1998). Using spreadsheets to implement the one-parameter item response theory (IRT) model. Paper presented at the Annual Meeting of the Southwestern Psychological Association, New Orleans.
[7] Lee, Y., Chon, Y. V., & Shin, D. (2012). Vocabulary size of Korean EFL university learners: Using an item response theory model. English Language & Literature Teaching, 18(1), 171–195.
[8] Moore, M., & Gordon, P. C. (2015). Reading ability and print exposure: Item response theory analysis of the author recognition test. Behavior Research Methods, 47(4), 1095–1109.
[9] Hambleton, R. K., & Slater, S. C. (1997). Item response theory models and testing practices: Current international status and future directions. European Journal of Psychological Assessment, 13(1), 21–28.
[10] Yamakawa, K., Sugino, N., Ohba, H., Nakano, M., & Shimizu, Y. (2008). Acquisition of English grammatical features by adult Japanese EFL learners: The application of item response theory in SLA research. Electronic Journal of Foreign Language Teaching, 5(1), 13–40. Centre for Language Studies, National University of Singapore.
[11] Nix, J.-M. L., & Tseng, W.-T. (2014). Towards the measurement of EFL listening beliefs with item response theory methods. International Journal of Listening, 28(2), 112–130. https://doi.org/10.1080/10904018.2013.872990
[12] He, L., & Min, S. (2016). Development and validation of a computer adaptive EFL test. Language Assessment Quarterly. https://doi.org/10.1080/15434303.2016.1162793
[13] McCarron, S. P., & Kuperman, V. (2021). Is the author recognition test a useful metric for native and non-native English speakers? An item response theory analysis. Behavior Research Methods, 53, 2226–2237. https://doi.org/10.3758/s13428-021-01556-y
[14] Hùng, T. N. (2023). The predictive validity of standardized tests of English proficiency in Vietnam (VSTEP). Tạp chí Khoa học Đại học Tân Trào, 9(2). https://doi.org/10.51453/2354-1431/2023/917
[15] Ministry of Education and Training. (2015). Document application guidelines: Format of test questions to assess English proficiency from level 3 to level 5 according to the 6-level Foreign Language Competency Framework for Vietnam (Decision No. 730/QD-BGDDT, dated March 11, 2015). Hanoi.
[16] Wu, M. L., Adams, R. J., & Wilson, M. (1988). ACERConquest [Computer software]. Hawthorn, Australia: ACER.
[17] Bock, R. D., & Aitkin, M. (1981). Marginal maximum likelihood estimation of item parameters: Application of an EM algorithm. Psychometrika, 46(4), 443–459. https://doi.org/10.1007/BF02293801
[18] Bachman, L. (1990). Fundamental considerations in language testing. Oxford University Press.
[19] Sireci, S. G. (2009). Packing and unpacking sources of validity evidence: History repeats itself again. In R. W. Lissitz (Ed.), The concept of validity (pp. 19–39). Information Age Publishing.
