FINE-TUNING DOMAIN-SPECIFIC PRETRAINED TRANSFORMERS FOR AUTOMATED SHORT ANSWER GRADING AND FEEDBACK GENERATION
DOI:
https://doi.org/10.18173/2354-1059.2026-0030Keywords:
automatic short answer grading, MathBERT, pretrained transformers, feedback generation, educational assessment, model generalizationAbstract
Automatic Short Answer Grading (ASAG) can return immediate, consistent scores and explanations at a scale that individual instructors cannot match, yet a single general-purpose encoder cannot produce open-ended feedback at all. This paper fine-tunes separate pretrained models for each task: MathBERT, a mathematics-education encoder (110M parameters), for binary grading, and FLAN-T5-base, an instruction-tuned encoder-decoder (248M parameters), for feedback generation, reporting the Matthews Correlation Coefficient (MCC) alongside accuracy and macro-F1 to ensure reliable evaluation under class imbalance. We evaluated the same experimental pipeline on BERT-base and SciBERT across three seeds to test whether mathematics-specific pretraining improves performance. It does not: averaged over seeds, MathBERT ranks last of the three on five of six test splits and ties on the sixth, and on the LaTeX-dense PRM800K dataset, where it should demonstrate the strongest performance, it trails BERT-base by 0.034 in macro-F1 (0.599 vs. 0.634). Performance is usable on answers to seen questions (macro-F1 0.864 on Beetle and 0.764 on SciEntsBank) and collapses on unseen questions and domains (0.508-0.555). A zero-shot Qwen2.5-1.5B-Instruct baseline outperforms the fine-tuned encoders precisely where they collapse, surpassing them by 0.114 and 0.105 macro-F1 on unseen questions and unseen domains while losing by up to 0.329 on seen questions. Additionally, it outscores the fine-tuned feedback branch (ROUGE-L 23.82 vs. 18.75). Composing the two branches end to end on PRM800K routes 67% of explanations to steps that were already correct. These findings suggest that, for these benchmarks, the choice of pretraining corpus matters far less than the difficulty of the generalization setting.
References
[1] M. O. Dzikovska, R. D. Nielsen, C. Brew & et al., “SemEval-2013 Task 7: The joint student response analysis and 8th recognizing textual entailment challenge”, Proceedings of the Second Joint Conference on Lexical and Computational Semantics (*SEM 2013), Atlanta, GA, USA, pp. 263-274, 2013.
[2] J. Devlin, M. W. Chang, K. Lee & K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding”, Proceedings of NAACL-HLT 2019, Minneapolis, MN, USA, pp. 4171-4186, 2019.
[3] OpenAI, “GPT-4 technical report”, arXiv:2303.08774, 2023.
[4] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix & et al., “LLaMA: Open and efficient foundation language models”, arXiv:2302.13971, 2023.
[5] A. Vaswani, N. Shazeer, N. Parmar & et al., “Attention is all you need”, Advances in Neural Information Processing Systems 30 (NeurIPS 2017), Long Beach, CA, USA, pp. 5998-6008, 2017.
[6] I. Beltagy, K. Lo & A. Cohan, “SciBERT: A pretrained language model for scientific text”, Proceedings of EMNLP-IJCNLP 2019, Hong Kong, China, pp. 3615-3620, 2019.
[7] J. Lee, W. Yoon, S. Kim & et al., “BioBERT: A pre-trained biomedical language representation model for biomedical text mining”, Bioinformatics, vol. 36, no. 4, pp. 1234-1240, 2020.
[8] J. T. Shen, M. Yamashita, E. Prihar & et al., “MathBERT: A pre-trained language model for general NLP tasks in mathematics education”, presented at the NeurIPS 2021 MATHAI4ED Workshop, 2021.
[9] C. Raffel, N. Shazeer, A. Roberts & et al., “Exploring the limits of transfer learning with a unified text-to-text transformer”, Journal of Machine Learning Research, vol. 21, no. 140, pp. 1-67, 2020.
[10] H. W. Chung, L. Hou, S. Longpre & et al., “Scaling instruction-finetuned language models”, Journal of Machine Learning Research, vol. 25, no. 70, pp. 1-53, 2024.
[11] D. Hendrycks, C. Burns, S. Kadavath & et al., “Measuring mathematical problem solving with the MATH dataset”, Proceedings of the NeurIPS 2021 Datasets and Benchmarks Track, 2021.
[12] H. Lightman, V. Kosaraju, Y. Burda & et al., “Let's verify step by step”, Proceedings of the International Conference on Learning Representations (ICLR), Vienna, Austria, 2024. github.com/openai/prm800k
[13] C. Sung, T. Dhamecha, S. Saha & et al., “Pre-training BERT on domain resources for short answer grading”, Proceedings of EMNLP-IJCNLP 2019, Hong Kong, China, pp. 6071-6075, 2019.
[14] L. Camus & A. Filighera, “Investigating transformers for automatic short answer grading”, Artificial Intelligence in Education (AIED 2020), Part II, Ifrane, Morocco, pp. 43-48, 2020.
[15] N. Reimers & I. Gurevych, “Sentence-BERT: Sentence embeddings using Siamese BERT-networks”, Proceedings of Conference on Empirical Methods in Natural Language Processing - International Joint Conference on Natural Language Processing, Hong Kong, China, pp. 3982-3992, 2019.
[16] S. Bonthu, S. R. Sree & M. H. M. K. Prasad, “Automated short answer grading using deep learning: a survey”, Machine Learning and Knowledge Extraction (CD-MAKE 2021), pp. 61-78, 2021.
[17] R. Ferreira Mello, C. Pereira Junior, L. Rodrigues & et al., “Automatic short answer grading in the LLM era: does GPT-4 with prompt engineering beat traditional models?”, Proceedings of the 15th International Learning Analytics and Knowledge Conference (LAK 2025), Dublin, Ireland, 2025.
[18] P. Lai, K. Zhang, Y. Lin & et al., “SAS-Bench: A fine-grained benchmark for evaluating short answer scoring with large language models”, arXiv:2505.07247, 2025.
[19] L. Cong, L. Hammerla, S. Hahn, S. Gombert, H. Drachsler & U. Kroehne, “Automatic short answer grading with LLMs: From memorization to reasoning”, Proceedings of the 16th International Learning Analytics and Knowledge Conference (LAK 2026), Bergen, Norway, 2026.
[20] M. Zhang, S. Baral, N. Heffernan & A. Lan, “Automatic short math answer grading via in-context meta-learning”, Proceedings of the 15th International Conference on Educational Data Mining (EDM 2022), Durham, UK, pp. 102-113, 2022.
[21] R. D. Nielsen, W. Ward, J. H. Martin & M. Palmer, “Annotating students' understanding of science concepts”, Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC 2008), Marrakech, Morocco, 2008. Release used: huggingface.co/datasets/nkazi/SciEntsBank
[22] M. O. Dzikovska, D. Bental, J. D. Moore, N. B. Steinhauser & G. E. Campbell, “Intelligent tutoring with natural language support in the BEETLE II system”, Proceedings of the Fifth European Conference on Technology Enhanced Learning (EC-TEL), Barcelona, Spain, pp. 620-625, 2010. Release used: huggingface.co/datasets/nkazi/Beetle
[23] M. O. Dzikovska, N. Steinhauser, E. Farrow & et al., “BEETLE II: Deep natural language understanding and automatic feedback generation for intelligent tutoring in basic electricity and electronics”, International Journal of Artificial Intelligence in Education, vol. 24, no. 3, pp. 284-332, 2014.
[24] T.-Y. Lin, P. Goyal, R. Girshick, K. He & P. Dollár, “Focal loss for dense object detection”, Proceedings of the IEEE International Conference on Computer Vision (ICCV), Venice, Italy, pp. 2980-2988, 2017.
[25] I. Loshchilov & F. Hutter, “Decoupled weight decay regularization”, Proceedings of the International Conference on Learning Representations (ICLR), New Orleans, LA, USA, 2019.
[26] C.-Y. Lin, “ROUGE: A package for automatic evaluation of summaries”, Text Summarization Branches Out (ACL Workshop), Barcelona, Spain, pp. 74-81, 2004.
[27] A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu & et al., “Qwen2.5 technical report”, arXiv:2412.15115, 2024.
[28] J. Cohen, “Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit”, Psychological Bulletin, vol. 70, no. 4, pp. 213-220, 1968.
[29] B. W. Matthews, “Comparison of the predicted and observed secondary structure of T4 phage lysozyme”, Biochimica et Biophysica Acta, vol. 405, no. 2, pp. 442-451, 1975.
