Insights and resources to support better data analysis in education
By Stuart Grey
The most common and popular strategy for assessing student feedback is to use quantitative data (e.g., course grades, which reflect the student's final opinion on the quality of instruction). However, at Student Voice, we feel that qualitative, free-text, comments from students can provide a level of detail and insight that assists lecturers and university management in understanding what the underlying issues and areas for improvement are.
Qualitative and quantitative data can both be used for reporting purposes in order to provide a more thorough representation of the information. Qualitative data should not replace quantitative data, but instead should be included with it in order to get a better picture of what students are thinking.
The aim of this research is to find ways in which universities can make interpreting student feedback easier. The paper discusses a university's use of an automated text analysis software to undertake the examination of student comments to better understand what students are saying and how academics can grapple with this valuable information. The paper focuses on the journey in the implementation of a suitable strategy for the text analysis of student comments collected in standardised unit evaluations.
This initiative sought to develop a system that would systematically evaluate the quality of learning and teaching in a university. The project aimed to provide a more effective, timely and thorough interpretation of the qualitative feedback received from standardised unit surveys.
There are often challenges in evaluating large numbers of student comments. This study trialed the use of text analysis tool to summarise student comments for selected units with very large enrolments. The study adopted a descriptive approach which assisted in understanding contextual issues, identifying key elements and to trial processes aimed at addressing the challenges involved.
The value of text analysis is that it simplifies the process by which qualitative data can be summarised, enabling academics to quickly gain an understanding of what comments are associated with particular themes. The potential limitation of this process is that it may not be representative as a result of the automatic classification.
The process of text analysis for feedback is used to find patterns in student responses about their university experiences. The decsion of which software or service to use is a key decision.
"The software packages are not easily interchangeable due to the different ways in which they work with text data. The decision on which package to use would depend on the nature of the research being undertaken."
The authors finally concluded that as the data complexity and heterogeneity grow, it becomes more difficult to make a simple decision about which tool is best for your needs. The right question could be what you want out of the tool rather than what tools are available.
The study found that, as the literature posits, there is considerable value in using automated text analysis to support the appraisal of comments students have made about their experiences. This particularly applies when the survey data are relatively large, such as in units which have large numbers of students.
The use of an automated process to reduce the time spent on text analysis does not however necessarily produce reports that are more meaningful or detailed when considering the categories applied. The adoption of a standardised dictionary may also present information in broad categories, hence the dictionary/lexicon should be continually refined to become more accurate, thorough and comprehensive.
This study aimed to explore the use of text analysis software for investigating students' experience in university courses. It has found that despite some limitations, the use of a software for text analysis can contribute to a fuller, more nuanced picture of students' experiences of learning and students' suggestions for improving teaching practices.
The introduction of the text analysis tool for qualitative data is an innovative approach to generating reports that can be used by stakeholders in their efforts to assess and improve quality assurance and enhancement processes. Further investigation is necessary to determine the usefulness of such reports to the stakeholders of the institution.
Overall we feel this study supports Student Voice's mission to improve the analysis of student free-text comments to make sure we are measuring both how effective our teaching is and the quality of the student experience.
[Paper Source]: Elizabeth Santhanam, Bernardine Lynch and Jeffrey Jones "Making sense of student feedback using text analysis – adapting and expanding a common lexicon"