AI learning surveys should measure adaptation, critique, and reflection, not just tool use
Updated Jun 14, 2026
student voicefeedbackUniversities are starting to ask students about Generative AI, but many surveys still measure usage more easily than judgement. That is why Yun Dai, Suya Liu, Sihan Zhou, Sichen Lai, Ang Liu, and Cher Ping Lim's Computers & Education paper, "Redefining and Measuring Student Agency in AI-assisted Learning: Development and Validation of the Agentic Engagement with AI (AE-AI) Scale", matters for teams using student voice to understand AI-assisted learning. If institutions want evidence they can act on, they need something better than a blunt question about whether students used AI.
Context and research question
Many higher education surveys now ask about AI in ways that flatten important differences. A student who uses GenAI to test an idea, challenge an explanation, compare sources, and refine their own reasoning is doing something very different from a student who uses it passively or uncritically. Yet those behaviours often disappear inside a single metric such as use, usefulness, or confidence.
Dai and colleagues argue that this is a measurement problem as much as a technology problem. Their starting point is that AI systems are no longer experienced only as static tools. Because they respond, suggest, and adapt, students' agency becomes more relational than many older instruments assume. The study therefore asks a practical question with direct relevance for UK higher education teams: if universities want to understand how students are learning with AI, what exactly should they measure?
The methodology is one reason the paper is useful. The authors generated an initial pool of 28 items from student interviews with 26 participants, then refined and validated the scale through expert review, exploratory factor analysis with 340 students, confirmatory factor analysis with 256 students, and criterion validity checking. That makes this a survey-development paper, but one grounded in student accounts first rather than imposed categories.
Key findings
The paper's core argument is that older ways of measuring student agency are now too narrow for AI-assisted learning. As the abstract explains, many existing instruments still treat agency as students exercising control over an external tool or environment. The authors argue that this misses the more reciprocal reality of working with GenAI, where students shape the exchange but are also shaped by what the system returns.
The validated AE-AI scale ends with 16 items across four factors: Adaptive Direction, Critical Integration, Cross-Source Inquiry, and Reflective Calibration. Even from the factor labels alone, the practical message is clearer than a headline "AI use" score. As an inference from those labels, the scale is trying to capture whether students steer AI use deliberately, test outputs critically, compare them against other sources, and adjust their approach reflectively rather than treating the tool as an unquestioned answer machine.
The paper sums up that shift neatly:
"a process of human-AI coagency grounded in adaptability, epistemic responsibility, and distributed inquiry"
That matters because it changes what good evidence looks like. If institutions only ask whether students are using AI, they learn almost nothing about quality of use. A higher score could mean curiosity and critical engagement, but it could equally mean dependence, confusion, or over-reliance. This study suggests that AI-related student insight becomes more useful when it separates those patterns rather than collapsing them.
The authors also give universities a more defensible way to connect closed-question data with open comments. The four-factor structure offers language for analysing what students are actually describing in their own words: are they using AI to orient themselves, to cross-check, to refine, or simply to outsource judgement? For teams collecting AI-related feedback, that is a more actionable frame than generic optimism or anxiety measures alone.
Practical implications
For UK higher education teams, the first implication is to rewrite AI survey questions so they measure behaviour with more precision. Instead of asking only whether students use AI, ask whether they use it to explore options, challenge interpretations, compare sources, refine drafts, or reflect on their own understanding. That produces evidence you can actually use in module design, academic skills support, and AI policy, which is the main benefit.
Second, universities should pair structured AI items with well-designed open-text prompts. A scale can tell you whether students report adaptive or critical use; comments explain where that pattern shows up, why it varies by course, and what support students think they need next. This is where a shared vocabulary helps. If teams are adding AI questions to module evaluations or pulse surveys, our student feedback analysis glossary and NSS open-text analysis methodology are useful starting points. The benefit is a clearer bridge between survey data and practical action.
Third, institutions should treat AI-related student voice as an evidence-quality issue, not just a policy issue. If comments suggest students are using AI mainly because instructions are unclear, feedback is hard to access, or they are unsure how to judge source quality, the intervention may need to sit in teaching design rather than AI rules alone. Student Voice Analytics fits naturally here because it helps universities group recurring themes in AI-related comments consistently, rather than relying on ad hoc reading or generic LLM workflows. The payoff is a more defensible reading of what students actually mean.
The broader takeaway is straightforward. If universities want to understand AI in the student experience, they need measures that recognise agency as something more demanding than access or adoption. Better questions produce better evidence, and better evidence leads to better decisions.
FAQ
Q: How should a UK university apply this paper when reviewing AI use in teaching and learning?
A: Start by auditing your existing survey or evaluation items. If they ask only about frequency of use or general usefulness, add prompts that distinguish between exploratory use, critical checking, cross-source comparison, and reflective revision. Then pair those items with one open-text question so students can explain where AI is helping and where it is making judgement harder.
Q: What should institutions keep in mind about the methodology of this study?
A: This is a scale-development study, so its strength lies in construct building and validation rather than sector benchmarking. The authors began with student interviews, then tested the resulting items through exploratory and confirmatory factor analysis, which is a solid route for instrument design. But institutions should still validate how well the framework travels into their own context, especially across disciplines and local AI policies.
Q: What is the broader implication for student voice?
A: It suggests that AI-related student voice should move beyond simple attitude polling. Universities need to understand whether students are using AI in ways that deepen judgement, widen inquiry, and support reflection, or whether the technology is becoming a substitute for confidence and clarity that teaching should already provide. That makes AI feedback a live part of student experience evidence, not a separate technical sideline.
References
[Paper Source]: Yun Dai, Suya Liu, Sihan Zhou, Sichen Lai, Ang Liu, Cher Ping Lim "Redefining and Measuring Student Agency in AI-assisted Learning: Development and Validation of the Agentic Engagement with AI (AE-AI) Scale" DOI: 10.1016/j.compedu.2026.105687
Request a walkthrough
Book a free Student Voice Analytics demo
See all-comment coverage, sector benchmarks, and reporting designed for OfS quality and NSS requirements.
- All-comment coverage with HE-tuned taxonomy and sentiment.
- Versioned outputs with TEF-ready reporting.
- Benchmarks and BI-ready exports for boards and Senate.
UK-hosted · No public LLM APIs · Same-day turnaround