An Empirical Analysis of LLM Usage in Software Engineering Course Projects

Abstract

Recent advancements in Large Language Models (LLMs) have had a substantial impact on software engineering research and practice, as well as on software engineering education. In Software Engineering (SE) education, prior studies have mainly examined students’ LLM usage through surveys, interviews, or short-term observations, offering limited visibility into their actual development activities. As a result, it remains unclear how students’ perceptions of LLM usage match their actual development practices. To address this gap, we analyze course project data collected over three years from 158 students enrolled in an upper-level SE course, together with students’ experience-sharing presentations, survey responses, and activity logs. We examine LLM integration by jointly analyzing students’ perceived usage and usefulness from surveys and presentations, their actual development activities captured through VS Code activity logs, and their project performance based on evaluation results across different levels of LLM support. By combining these analyses, we connect students’ reported LLM usage with their actual development activities and assess its relation to learning outcomes in SE course projects. Our results show that students utilized LLMs frequently for code-related tasks and generally found them helpful, with frequent usage increasing over the course project while perceptions shifted as project complexity grew. Activity log analysis revealed divergent patterns in LLM usage, reflecting both increasing dependency and deliberate efforts to maintain development independence. Across three years, integrating LLM support did not lead to statistically significant differences in final project achievement as measured by requirement satisfaction, although students with LLM support produced substantially larger software systems and lower-performing teams showed mild benefits, suggesting that LLMs may help reduce the influence of programming proficiency on project outcomes. Our findings highlight both the benefits and tensions of LLM integration in SE course projects and suggest that future SE course designs may incorporate more complex and challenging SE tasks beyond implementation-focused requirements, while also providing explicit education on prompt engineering, supporting students’ code comprehension of rapidly generated LLM-assisted code, and helping students avoid excessive dependency on LLM tools during development.

The authors' abstract, as published at the source. ACM Transactions on Computing Education, 2026 · DOI ↗