Putting NSS comments in sector context

Student Voice Weekly, episode 36, with Dr Stuart Grey.

Dr Stuart Grey explains how Student Voice AI compares positive and negative NSS feedback with corresponding sector benchmark sets using the same deterministic models. He follows sentence-level categories into whole comments, then explores discipline and demographic differences, including nursing placement experiences.

Chapters

00:00 - Benchmarking free-text student comments
00:51 - Why customers use NSS sector analysis
01:38 - Sentence-level categories and whole comments
02:36 - Positive and negative feedback benchmarks
03:20 - Consistent deterministic models
03:47 - Reading the feedback behind a comparison
05:17 - Discipline differences: nursing and engineering
06:17 - Comparing relevant discipline benchmarks
07:09 - Demographic views
08:11 - Further information and subscribe

Features and further reading

The assessment feedback comparisons in this episode are illustrative. Reading the original survey comments helps colleagues understand the experiences behind a benchmark difference. Relevant reference groups, compatible collection periods and sufficient group sizes matter when interpreting comparisons.

Subscribe to The Student Voice Weekly. Browse the podcast.

Transcript

Hi, and welcome to Student Voice Weekly. I'm Dr Stuart Grey, founder of Student Voice AI, and this week I'd like to talk about benchmarking of our free-text comments and particularly how it can help us understand our own institution in the context of the wider sector.

So last week we looked at school and programme-level reports and how local teams can use the same evidence base as a wider institutional research team. Benchmarking sector surveys such as NSS gives us a useful next step.

So universities already use the quant results to understand how their students' experiences compare with the wider sector. The comments add detail and richness to this about what the students actually mean, what the sentiment is around certain parts: what worked well, what was difficult, and in particular, which parts of the experience they chose to describe in their comments and which ones they left out.

One of the key things our customers use Student Voice AI for is this NSS sector analysis. They want to understand how students feel about different parts of their experience, how that compares with the sector, and where the picture changes between disciplines or student groups. That gives institutional research teams something useful to take into a discussion with colleagues to actually make the changes to improve our provision.

The usefulness of NSS here is the shared survey context across all universities in the UK. It gives us a practical basis for looking at the same areas of student experience across institutions with an appropriate comparison and reference group to use. We can then examine the positive and negative feedback within each of those areas in much more detail than we get with the quant data.

Before we get into that comparison, it's useful to explain how we analyse the comments themselves. So we work sentence by sentence. Each sentence can be assigned multiple different categories because students can describe more than one aspect of their experience in a single sentence.

So imagine a sentence about last-minute timetable changes and how poorly those changes were communicated. It can receive the relevant categories for both the organisation of the course and the communication around it.

The next sentence might praise a helpful tutor. These are different parts of the same student's response, and we want to keep that detail.

So what we do is we analyse them sentence by sentence, then we roll up all these sentence-level category assignments to the whole comment. So a comment can and does appear under many different categories, including more than one category coming from the same sentence.

So we retain the connection with the full response, so colleagues can read what the student actually said in context.

Topic volumes help us see which areas students are discussing. Are they talking more about A or B? The main value I want to focus today on, though, is on the sentiment within those comments. That's where real richness comes through. What students are describing positively, what they describe negatively, and how that compares with the other users in the NSS benchmark.

We can split out the positive and negative feedback within a topic. We can then compare the positive feedback with the positive benchmark set and the negative feedback with a negative benchmark set and see where we differ.

Both views come from the same underlying comment analysis, and the institutional data and the benchmark data are categorised using exactly the same deterministic models, so we can have real trust in the analysis.

That consistency is really important. We want to use the same category definitions and the same sentiment models on both sides.

So a deterministic model gives the same result for the same input when the model is unchanged. So that gives us confidence that we're making the comparison on a consistent analytical basis with the same method applied to the local comments and the reference data, and importantly, all historical data too. So we use the same models across all our datasets.

So think about assessment feedback. Positive comments might describe detailed explanations that help students improve their work. Negative comments might describe feedback arriving too late or leaving students unsure what to do next. Looking at those views separately helps us understand what is being valued and what is causing difficulty.

Suppose as an illustration that your positive feedback picture compares favourably with the relevant NSS benchmark, while the negative picture still deserves attention. So reading across the comments might show that students value the quality of individual explanations, but experience problems with timing or consistency of feedback.

Those findings can sit together. They give colleagues a much clearer basis for discussing what to preserve and what importantly to improve.

A student can also praise one aspect and criticise another within the same response. That's why the sentence-level work really matters. Helps us retain those different experiences when we build the wider picture, and then return to the original comment when we need to understand how they connect.

So we use these benchmarks to allow us to find areas that we're interested in and then dive into the actual comments in those areas.

For an institutional research or surveys team, the useful question becomes: Where does a sentiment differ from the relevant benchmark set, and what are students actually describing within that difference? The comparison gives us a direction to investigate. Reading across the relevant comments helps us explain it. This includes where experiences are mixed or vary between subgroups of students.

And this is where the discipline splits become really useful. So students on different courses have quite different experiences and different things they want to talk about. An overall university comparison can bring together subjects where the priorities and the structure of the course are widely different.

So nursing is a clear example. So placements obviously play a major part of the student experience in nursing courses, so they become a major part of what the nursing students discuss, both positive and negative. So students might talk about support on placement, supervision, how placements are organised, or how the experience connects with what they have been taught. Both positive and negative accounts of those experiences matter.

That won't necessarily be the same focus for, say, engineering students. They may be talking about laboratory access, product facilities, or equipment. Engineering students can have placements too, but we shouldn't assume that placement experience has the same role, for instance, across those different programmes as it does in nursing.

So when we do our comparisons with the benchmark, we can compare nursing students with us versus nursing students elsewhere.

So if we understand the sentiment in nursing placement comments a relevant nursing reference gives us a much more useful starting point than a figure combining every subject.

So we can compare the positive picture with the relevant positive benchmark and the negative picture with the corresponding negative benchmark within that discipline.

Then again, we can read the local comments to understand the particular experiences behind each aspect that we're looking at.

The same principle applies when looking across schools within a university. Different subject mixes can shape an institutional sort of outcome. A discipline view helps colleagues see whether a finding reflects something specific to their area or a pattern that appears more widely in comparable courses.

Alongside those subject differences, we can also split analysis by demographic group.

This might include age, ethnicity, disability, study mode. We use these fields to look at how the themes and sentiment vary across different student groups we have represented in the data.

And again, we're comparing like for like. For example, a broadly positive picture of academic support might contain different experiences for students who report a disability.

Reading the relevant comments could help a team understand whether the differences are about arranging support, accessing it in practice or something else entirely.

So that's just one potential question to investigate. There are many more, I'm sure you can think of them, about what different subgroups experience, and our dataset allows you to understand how that subgroup at your university experiences things compared to that same subgroup elsewhere.

So thanks for listening again to Student Voice Weekly. If you'd like any more information or sign up for the newsletter, please go to Student Voice AI. And if you found this at all useful, please share with any colleagues across the sector. Thank you.