Where GenAI fits in student comment analysis

QAA and Swansea University have published a practical approach to using generative AI for a first pass over free-text student survey comments. Dr Stuart Grey examines the safeguards author Sophie Leslie recommends and asks when universities need a more reproducible analytical foundation.

The episode explains why Student Voice AI separates deterministic machine-learning categorisation and sentiment analysis from locally run LLM summarisation, and why that distinction matters for comparisons across cohorts, faculties and years.

In This Episode

• What Swansea University's GenAI-assisted comment-analysis pilot involved.
• Why QAA recommends consent checks, anonymisation, controlled infrastructure and human validation.
• What the Swansea example does and does not establish about effectiveness.
• The difference between exploratory thematic work and repeatable institutional analytics.
• Why Student Voice AI uses deterministic models for categorisation and sentiment.
• Where locally run LLM summarisation fits after the analysis.
• Practical stability and provenance checks for university pilots.

About This Recording

Recorded by Dr Stuart Grey for Episode 28 of Student Voice Weekly. The transcript has been lightly edited for clarity while preserving the substance and natural delivery of the recording.

Sources

• Faster thematic analysis of student survey comments, by Sophie Leslie of Swansea University
• QAA sets safeguards for GenAI student feedback analysis
• Student Voice Analytics versus generic LLMs
• Student Voice Analytics

Practical Takeaway

Decide whether the task is exploratory or repeatable. For an exploratory first pass, test GenAI with QAA's safeguards. For reporting, benchmarking or longitudinal comparison, use a method that can reproduce its categories and sentiment, preserve a route back to the evidence, and maintain appropriate data residency.

Subscribe

Listen to and subscribe to Student Voice Weekly, or receive the written newsletter each Friday through The Student Voice Weekly newsletter.

Transcript

Hi, and welcome to Student Voice Weekly. I'm Dr Stuart Grey, founder of Student Voice AI.

And this week I'd like to focus on one question: where should generative AI actually sit in the analysis of student comments?

I still teach part-time at the University of Glasgow, and I spend a lot of my working week looking at student feedback data. So this is something I come at from both sides. I care about making analysis faster, because university teams simply don't have unlimited time to look at these comments. But I also care about whether the results can be checked, repeated and, ultimately, trusted.

The piece I read this week was published by the QAA as one of its Quality Quick Wins. It was written by Sophie Leslie, who is Education Quality, Policy and Governance Manager at Swansea University, and it describes Swansea's pilot use of generative AI to analyse free-text student survey comments.

It is a useful piece. It's short, practical and much more careful than the sort of advice that simply says: put your comments into an AI tool and ask for some themes.

The problem Sophie starts with will be familiar to almost every survey or quality team. Free-text comments are rich, but they take a lot of time to read, code and categorise. So when thousands of comments arrive together, people may skim them, analyse a sample, or finish the work too late for it to shape any real enhancement discussion.

The Swansea approach uses generative AI for a first-pass thematic analysis. The tool is asked to identify recurring themes, sentiment and specific challenges, then return a structured report with theme names, comment counts, sentiment and verbatim quotations.

QAA's suggested prompt is quite disciplined in this case. It tells the tool not to paraphrase the quotations, to add no interpretation beyond the requested themes and quotes, and to keep anything that does not fit in an uncategorised group.

So far, so good. And it is really important to have all those guidelines in place, because an uncategorised group is also clear evidence. It shows you what the frame did not capture. If the system quietly forces every comment into the nearest available heading, the final distribution can look really tidy while hiding the fact that students were talking about something entirely different.

The safeguards in the QAA piece are also quite strong. They say to check that individual consent or the privacy notice covers AI-assisted analysis. You've got to anonymise or pseudonymise comments. You've got to use an approved, institution-controlled environment, which is super important. Complete a Data Protection Impact Assessment. All these are really important things.

You've also got to be transparent with students, validate the groupings against source comments-ideally with more than one reviewer-and treat this whole process as a first draft rather than a final judgement.

QAA also says very clearly that you never paste student comments into a public chatbot. I think that bears repeating, because I've seen it happen a lot of times and we really can't be doing it. Student comments can contain names, health information, accounts of harassment, complaints about staff, or enough detail to identify someone. The fact that the comments arrive through a survey does not make them harmless or anonymous. You've got to be really careful with this data.

Swansea reports that the pilot produced near-immediate initial analysis of all the comments, identified key themes and possible actions, and saved hours of effort. This is promising, but the article does not tell us the size of the dataset, which tool or model was used, how accuracy was measured, or how stable the output was across repeated runs.

This is really important. It's only genuinely useful if you can do it again and get the same answer, which many prompt-based generative AI processes will struggle with.

I think this is where universities need to make a really key distinction. There's a difference between using generative AI to explore a set of comments and using an analytical method that needs to support comparisons across faculties, student groups or years. They are two very different things.

For a small, low-risk dataset, a well-governed generative first pass like this may be a useful way to start. You can read the proposed themes, compare them with the source material, and decide whether there's something worth investigating. As a small case-study research approach, it's good.

But if those categories and sentiment scores are going into institutional reporting, benchmarking or year-on-year trend analysis, repeatability matters much more-and you can't use this approach in a repeatable way.

At Student Voice AI, we deliberately separate the analytical foundation from the written summary we produce. All categorisation and sentiment analysis is done with deterministic machine-learning models. LLMs are not used for those tasks.

With the same input, model version and settings, the analytical output is exactly the same. Give it the same sentence and it will give you the same output. Every valid comment is analysed. Again, we can guarantee that, whereas with generative AI you cannot.

Where a comment contains several ideas, the analysis works at sentence level and can apply multiple categories per sentence rather than one broad label, which is something these generative AI approaches tend to do.

This stable, analytical, deterministic base really matters because universities want to compare things: year on year, between faculties, schools, CAH codes, or whatever it might be. Has the sentiment about feedback and assessment changed since last year? Is student support being discussed differently in one faculty than another? Does a category appear unusually often compared with the sector? Those sector comparisons are really useful.

You need to know that any change reflects the evidence rather than a slightly different prompt, a slightly different model, or a different version of the same model. These things get updated very frequently, with a generative system making a different choice one day compared with another.

We only use LLMs for summarisation after the categories and sentiment have been produced. The LLM's job is to help communicate a structured body of evidence. We have all that structure already and the LLM is just there to make a little bit of text from it. It does not create the category counts or sentiment results underneath. That's all done with our deterministic machine-learning models.

The summarisation models we use are run locally on Student Voice AI-owned hardware. We don't send comments to any public LLM services. UK customer data is entirely resident in the UK, and our Irish customer data is processed entirely in the EU environment.

We are very strong on data protection because that's a key risk with using these LLM tools: someone puts the comments into one of the public tools by accident and you have a major data breach at that point.

Our answer isn't that generative AI has no place in comment analysis. It's very useful for language-heavy tasks. The key thing is to give it the right job.

This is actually very close to the spirit of what QAA is warning: that GenAI themes are a first draft. Where I would go further is that human review does not by itself make a changing method reproducible. It's still non-deterministic. A reviewer can spot an obviously misplaced quotation, but it's much harder to see that hundreds of borderline comments have shifted subtly between categories A and B because the model or prompt behaved differently that day.

If I were evaluating a generative AI pilot, I'd therefore test stability as well as plausibility. Run the same comments more than once. Record the exact model version, prompt and settings used, and compare the category counts between runs. Look separately at the uncategorised material, and check performance for small cohorts and for comments containing several different issues. Again, these are things that generative AI can silently trip up on.

Finally, you want to make sure that any headline conclusion has a direct route back to the evidence.

I'd also ask what happens next. A fast thematic report only becomes useful when somebody owns a response. If it's not trusted, people won't be able to take ownership of it and actually make the change needed. People need to be able to see the underlying comments. They need to trust the process and understand that it is deterministic, solid and stable over time. If it's not, people will err on not trusting it.

The point is to support people talking to students, talking to other staff members, and help the right part of the university act on what students are actually saying.

My takeaway from this Swansea study is positive. QAA has set out a responsible starting point for a small-dataset exploratory study, which is a good use of the tool. The key thing is that it's not a good tool when you do large-scale analysis and want to compare against benchmarks or over time with a stable baseline.

If you are doing one of these small research projects with generative AI, I suggest you begin with a small dataset, make sure you establish consent and privacy, use a controlled environment, insist on verbatim evidence, and keep people responsible for any ultimate judgement that comes out of the system.

Thanks for listening. If this episode was useful, please share it with somebody working in quality, surveys or institutional research, and I'll talk again next week. Thanks.