Research on Education Text Analysis and Teaching Best Practice
Anosh Butt - Updated Mar 21, 2022
This case study investigates research conducted by academics and researchers from Newcastle University. The research investigates undergraduate report marking involving two assessors; however, marking models are not consistent as some institutions use two independent assessors and do not use the project supervisor (McQuade et al., 2020). The authors highlight that critics point out that the latter model may create a supervisor bias, while its supporters suggest that it ensures subject expertise, which is crucial in the assessment process. The undergraduate biosciences programs at Newcastle University utilise the supervisor as one of the report assessors, while the postgraduate programs do not. The research explores the impact of supervisor marking, compares undergraduate and postgraduate project marks between the period: 2011/12 – 2016/17, and analyses two different reconciliation processes quantitatively.
The authors present that the project marking data was associated with undergraduate and postgraduate biosciences programmes delivered between the academic years 2011/12 – 2016/17. During this period, the total number of undergraduate projects was 1460, with 897 relating to four academic years 2011/12 – 2014/15 (reconciliation method I), and 563 related to two years 2015/16 – 2016/17 (reconciliation method II). Postgraduate marking data related to period 2011/12 – 2015/16 and represented 894 project reports.
The undergraduate project reports were presented in scientific paper format and had a word limit of 5000 words, while the postgraduate project reports were presented in a thesis report format and had a word limit of 8000 words. The pass mark for the undergraduate and postgraduate projects were 40% and 50%, respectively. Project supervisors for undergraduate and postgraduate research projects involved up to 150 academics, spread across five different research institutes and three schools (McQuade et al., 2020).
The first reconciliation method consisted of discussions among the two markers, and they were required to submit an independent mark, supported by written comments. It was used for undergraduate report reconciliations between 2011/12 – 2014/15 and postgraduate reports during 2011/12 – 2015/16. The second reconciliation method comprised discussions between the two markers, and they were required to reconcile their marks, so they return a single agreed mark and one set of supporting comments. It was used for undergraduate report reconciliations between 2015/16 – 2016/17.
The research data were divided into three groups, which consisted of undergraduate project report marking 2011/12 – 2014/15, postgraduate project report marking 2011/12 – 2015/16, and undergraduate project report marking 2015/16 – 2016/17. The data was analysed using the difference between the grades awarded by markers one and two. For undergraduate reports, marker one was undergraduate supervisor and marker two was a faculty academic, and for postgraduate reports, marker one was an internal examiner with appropriate subject knowledge, and marker two was an external examiner with appropriate expertise.
The analysis of undergraduate marking data over a four-year period evidenced a mean difference between supervisors and faculty academic of +2.3% (McQuade et al., 2020). This evidence reflects independent double marking and supports an agreement between markers, although skewed towards supervisors, and shows that supervisors mark marginally higher than second markers. There were some instances, which showed differences between markers of up to 30%, and showed less robustness in marking consistency. These discrepancies were not limited to the supervisors’ marks.
An interesting observation regarding postgraduate dissertations is that with the same internal pool of academics as first markers, there were still discrepancies of up to 30%, although the internal markers did not know the postgraduate students. Factors contributing to the marking bias include the level of subject knowledge of the markers, report marking experience, and first and second marker subconsciously favouring certain elements of marking. It is important to note that the second markers of the undergraduate report and the external assessors of the postgraduate reports each marked more dissertations than the supervisors/internal markers, and the diversity of project topics that they had to grade was significant, which may compromise their subject knowledge and marking reliability.
The authors report that there is evidence that subject knowledge may not be essential for marking student dissertations, and these scenarios relate to the assessment of more general components such as document structure, presentation, style of writing, and referencing rather than scientific content. The authors also highlight that assessors without relevant subject expertise became more cautious and are less likely to use the full marking range, but on average, they marked significantly lower than those with subject-specific knowledge.
McQuade et al. (2020) acknowledge that reconciliation comments evidence content relating to student originality, breadth, and depth of scientific knowledge, independent critical thinking, and scientific argument. Finally, the authors comment that there is a pressure in higher education to move away from the undergraduate supervisor in the dissertation marking process, and the drive for change is due to the perceived supervisor leniency associated with student contact and familiarity caused by superior subject knowledge of the supervisor.
[Source Paper] McQuade, R., Kometa, S., Brown, J., Bevitt, D. and Hall, J., 2020. Research project assessments and supervisor marking: maintaining academic rigour through robust reconciliation processes. Assessment & Evaluation in Higher Education, 45(8), pp.1181-1191.