Evaluation of ChatGPT’s Implementation in Undergraduate Java Programming Training

Abstract

There is extensive research discussing the potentials and challenges of applying generative AI in educational practice, although empirical studies are rare, especially those integrating generative AI into specific subject learning. To fill this gap, this study conducted quantitative research to investigate the effectiveness of ChatGPT-centered Java Programming learning. Fifty-three undergraduate participants attended the study, divided into two groups over a seven-week learning intervention: the teacher-centered learning group and the ChatGPT-centered learning group. Pre- and post-knowledge tests, along with questionnaires assessing learning motivation and self-efficacy, were used to evaluate participants’ learning outcomes. The results indicated that, compared to the traditional learning method, while ChatGPT’s effects on knowledge assessments were modest, its impact on learning motivation and self-efficacy was significantly more pronounced in Java Programming education. This finding suggests that ChatGPT can be an effective tool in bolstering learners’ motivation and self-efficacy, regardless of the students’ initial proficiency level in programming. By comparing the results with related studies, we further explored how students’ learning motivation and self-efficacy could be enhanced by ChatGPT. Furthermore, we identified the research limitations in this study and suggested directions for future research. Specifically, we noted the need for more diverse and comprehensive assessments to fully capture learning outcomes and the importance of incorporating qualitative methods to gain deeper insights into learners’ experiences with ChatGPT-centered learning. Future research should also explore the long-term effects of using generative AI in education and its applicability across different subjects and educational contexts.