The comments your search words leave out

Dr Stuart Grey starts with a familiar workflow: finding relevant student comments in an Excel survey export. He compares the useful text-search and filtering controls in that workflow with the Student Voice review tools, explaining what meaning-based discovery, precise searches and sentence highlighting add to an institution-wide investigation.

In this episode:

Using Find, text filters and wildcard searches in an Excel export.
Why wording searches may leave differently expressed experiences outside the results.
Combining text and meaning-based matches to broaden discovery.
Switching to precise wording when a named phrase matters.
Understanding the effect of whole-comment exclusions.
Highlighting matching sentences while retaining the complete response.
Bringing year, topic and sentiment filters into the review workflow.
Keeping relevance, interpretation and prevalence distinct.

Chapters

00:00 - Searching comments in an Excel export
01:17 - What search words can leave out
02:09 - Combining text and meaning-based search
03:11 - Keeping the reviewer's judgement central
03:39 - Precise phrases and named services
04:18 - Whole-comment exclusions
05:03 - Matching sentences in the full comment
05:43 - Understanding why a result appeared
06:04 - Year, topic and sentiment filters
06:23 - What each approach supports
06:50 - Relevance and prevalence
07:11 - An exploratory approach and closing

Search tools

Transcript

Hello, and welcome to Student Voice Weekly. I'm Dr Stuart Grey, founder of Student Voice AI, and today I'd like to talk about searching student comments. I want to start with a familiar situation. You've got a survey export in Excel, and a colleague wants to know what students are saying about a particular issue. So I can see why this is a useful starting point for teaching teams and institutional researchers, as I've done it myself many times over the years. So the comments are in a file you can work with. You can read through the responses, keep notes, and bring relevant evidence into the discussion about a course or service.

For a straightforward search, you might use the Find tool in Excel to look for a word or phrase. You might put a text filter on the Comment column using something like Contains Feedback. If your export includes a year or programme column, you can narrow the rows further. That's a practical way to get to a set of comments you want to read.

Excel also gives you options such as wildcard searches, so you can allow for variations in the wording. Those built-in Find and filter tools are useful when you know what you're looking for. You're searching within the free text responses, but the search criteria still depend on the words or patterns that you specify.

So now imagine that the question is about feedback arriving too late to be useful. A search for feedback gives you a very wide range of responses. A search for late narrows things down differently, but a student might describe getting feedback after submitting the next assignment without using the word late. The experience could be relevant to the review and still sit outside that particular search we've done.

So you can respond to this by trying more terms and reading across the results and then more terms, et cetera. The difficulty is deciding which other phrases to try. As a comment set grows, you are managing a longer list of possible wording, checking repeated matches, and judging which responses address the same experiences. There is quite a lot of interpretation inside what initially looked like a simple search task.

That is a problem we have tried to address in the Student Voice AI suite of tools. We've kept the ability to search for particular words and added a way to search for similarity in meaning. The aim is to give colleagues a broader route into the student experience without requiring them to predict every phrase students might use.

In an ordinary search, we combine the exact text matches with the meaning-based matches, so the text matching remains useful for specific terms you've entered. The meaning-based part can bring in differently worded accounts that may be relevant to the same question. We combine those results into a ranked list for the reviewer to read and interpret as before.

Going back to the assessment example, the benefit is a way to explore accounts of feedback arriving after an opportunity to use it. You can describe the experience you're investigating and look at the language that comes back. That can help you discover wording you hadn't thought to search for, then refine and review with that better understanding.

The judgement obviously still sits with the person reading. Some matches will be more useful than others, and the connection in meaning always needs checking, of course. And you wanna be looking through a range of results, looking for accounts that add nuance to your first impression and adjust the questions and filters as necessary. The search gives us a way into the evidence with the original comment available to inspect, but this improved method allows more relevant comments to be surfaced.

But sometimes we need the opposite kind of control. We might be looking for a particular service name where the exact wording matters. Excel's text tools are useful for that, and we have kept that kind of precision in our approach too.

In the review tools, quotation marks around a phrase select a precise search, which means the meaning-based expansion is switched off. The benefit is being able to move between broad discovery and a specific wording check within the same review. If the first pass raises a question about a named service, you can investigate that phrase directly. You know that this search is using the exact text matches rather than widening the results to related meanings.

We also support exclusions, so you can put a minus sign before a word and comments containing that word are excluded from the search. For example, you might look for finance while excluding accommodation. The exclusion applies to the whole comment, so a response that mentions accommodation elsewhere will also be removed.

So obviously these tools have to be used with care. A financial difficulty, for example, may be closely connected with accommodation, but sometimes you're looking for comments about A and definitely not about B. So this lets us create broader and narrower sets and allows us to really tune the response to the system to what we need to get the actual comments we want.

There's also another practical issue when comments are very long or cover several topics. In a spreadsheet, you can read the complete response in its cell. In our review tools, we highlight the matching sentences within the full comment. This really helps the reviewer locate the relevant part within the larger comment while keeping the rest of the response there as context.

So imagine a student praising the person who eventually helped them while describing difficulty getting through to that person. We want the appreciation and the access problem to remain visible together. Highlighting a sentence helps somebody find the point quickly. The surrounding comment helps them decide what it means.

We also show where the result came through the exact text matching, the similarity in meaning, or both routes. That gives a reviewer an explanation of why it appeared, so that provenance. If a result seems unexpected, they can inspect it with that in mind. It helps make the search process something colleagues can question and understand and ultimately trust.

Filtering obviously remains useful whichever tool you use. In Excel, you work with the columns in your export. In our review tools, we've brought year, topic, and sentiment filters into the search workflow. That lets a reviewer move from an institution-wide theme into a more focused set of comments while retaining the search and the full response together.

For an institutional research team, the useful comparison is between the jobs each approach supports. Spreadsheet is flexible for organising and working with an export. Our search adds an extra route to differently worded experiences alongside the text controls and context needed to investigate them. Those capabilities are intended to reduce how much review depends on guessing the right vocabulary when searching for the right terms.

I would still keep relevance and prevalence separate. A ranked set of search results helps us find accounts to examine. To report how common an experience is, we need a defined population and consistent analysis across that set. Changing a search phrase and getting more matches tells us something about the search rather than demonstrating a change in the student experience.

So I begin with the experience we want to investigate and then start using the search tool as an exploratory tool to try and find out, okay, what are the different types of things students are talking about? What does it co-occur with? What is the context within the larger comments, et cetera?

So I hope you found that useful. You can find more information on the website Student Voice AI. And if you found this podcast good, please do share with colleagues. Leave the podcast a review and subscribe. Thank you very much for listening, and I'll speak to you again next week.

Book a demo or get in touch

The comments your search words leave out