Student evaluation scores are not automatically comparable across departments, programmes, or time

Updated Sep 10, 2026

Before comparing teaching-evaluation scores across groups, check whether the instrument supports that comparison. Edgar Valencia examines this question in a study of student evaluations of teaching.

What the study reports

Using data from a graduate school of education, the study tested measurement invariance across programme types, academic departments and terms. This concerns whether the measurement model functions consistently across the groups or occasions being compared.

The authors report no support for strict comparability across these variables. Valencia argues that comparability checks can inform responsible use of scores and improvement of evaluation instruments.

These findings do not justify treating every possible use of the instrument as invalid. Interpretation requires attention to the level of invariance tested and the proposed decision.

Implications for institutional review

As editorial recommendations, involve appropriate measurement expertise before using scores for consequential comparisons. Specify whether the question concerns change within a group, differences between groups or individual personnel decisions.

Comments may provide context and questions for investigation, but they do not automatically repair a non-comparable scale. They also require attention to who responded, what was asked and how the material was interpreted. Use each source for the claims it can support.

Document the limitations alongside any dashboard or report, so a convenient ranking does not become stronger evidence than the analysis permits.

Reference

Edgar Valencia. “On the comparability of SET scores: measurement invariance across programs, departments, and time.” Assessment & Evaluation in Higher Education. DOI: 10.1080/02602938.2025.2508754.

Corrected 7 September 2026: removed unsupported rankings of comparability problems and claims that comments provide a measurement remedy.