Student-support chatbots: useful feedback, uncertain headline figures

Updated Sep 07, 2026

Positive comments about a chatbot can be useful without proving that it improves learning or saves staff time. Kremantzis and colleagues studied a rule-based support chatbot in two UK Business and Economics units. Reading beyond the abstract reveals important qualifications to its headline figures.

What the evidence supports

The units enrolled 299 students; 84 answered a questionnaire and 34 reported using the chatbot. Among those 34, 44% said they found all the information sought, 47% only part and 9% none. Combining the first two groups as 91% obscures the distinction between complete and partial help.

All 15 respondents to the speed question valued the faster response. The question was restricted to students whose query had been resolved, so this is not a rating from every user.

There is also an unresolved reporting discrepancy: the abstract and conclusion give 59% successful interactions, while the results and Figure 3 classify 51.3% of 156 chats as successful. We therefore do not offer a single success-rate benchmark. The study did not establish causal improvements over existing support.

What a university could check locally

Our recommendation is to define success before launching a pilot. Separate finding a useful link, finding part of an answer and resolving a question completely. Count conversations separately from people, and show the denominator beside every percentage. Ask unsuccessful users what happened next as well as collecting satisfaction from those who received help.

A pilot could also track subsequent staff contact, unresolved questions and the time required to maintain information. These are proposed evaluation measures, not benefits demonstrated here. Keep an accessible route to human support and test whether students can find it without first completing a chatbot conversation. The governance checklist can support discussion of responsibilities and comment handling.

FAQ

Was this a generative AI assistant?

No. It used predefined decision paths and curated information. Results should not be transferred automatically to a system that generates new answers.

How much of the source was checked?

The abstract and accessible methods, results and concluding sections were reviewed. This is a limited-source review, with the source's inconsistent figures disclosed rather than resolved by assumption.

Reference

Kremantzis, M., Chondrogianni, A., & Essien, A. (2026). Evaluating the impact of AI chatbots on student support and engagement in UK higher education. Journal of Further and Higher Education.

Correction, 7 September 2026: added the survey denominators, distinguished partial help from full resolution, disclosed conflicting success figures and removed unsupported impact claims.