Findings from the TASC (Part I: Assessment)

An overview of the results of the ‘Assessment’ portion of the SU’s Teaching and Assessment Student Consultation (TASC) for Trinity 2020


In light of the COVID-19 pandemic, the University took the unprecedented step of sending students home and moving all of its teaching and assessment functions to ‘remote’ platforms for Trinity Term 2020.

In order to aid the work of Sabbatical Officers working with and lobbying the University during this period, Oxford SU has sought to better understand the ramifications of the many changes the shift to remote teaching and assessment could entail.

To this end, Oxford SU created the Teaching and Assessment Student Consultation (TASC), in collaboration with the Pro-Vice Chancellor for Education, Education Policy Support, the Student Registry, the Centre for Teaching and Learning, and other University stakeholders.

The focus of the TASC was to identify the problems that students might face in light of changing circumstances and to ensure that the University was aware of the effects its various potential policies would have on students.

The TASC received 5462 submissions before the deadline with many students writing passionately and in great detail. Collectively, SU staff and Sabbatical Officers read almost 2 million words written by students over the course of around a week and a half! For reference, the ‘Harry Potter’ book series contains about 1 million words. On top of this, dozens of students wrote personally to myself and my fellow Sabbatical Officers via email and social media and hundreds more affixed their names to open letters outlining their concerns.

The TASC has provided enormously useful insights on matters big and small. It has informed and continues to inform the work of the SU during these difficult and unexpected times. I have used it as the basis for my own arguments in conversations with top University officials and stakeholders across the collegiate University and its findings have informed policymaking around teaching and assessment in Trinity 2020.

I am very grateful to all the students who took the time to write submissions to the TASC and commend the tireless work of SU staff in amplifying the student voice.

 

Ray Williams

Vice-President (Access & Academic Affairs)


Who Responded?

The TASC attracted submissions from students on a broad range of courses. As is typical with SU initiatives, engagement was higher with the undergraduate student body but the TASC still managed to capture the views of postgraduates, including some postgraduates in teaching roles.

Although an overview of the entire cohort is given below, the structure of the TASC allowed for a fine granular analysis of the issues faced by students. TASC information broken down by Division was made available to Oxford SU’s Divisional Representatives to aid their work representing the academic interests of their peers at the divisional level. When working on policy-questions relating to Finals Papers, the views of the affected students were prioritised.  It is worth noting that the views of those students in their final year of study closely mirrored the views of the wider cohort.

 

 


Survey Structure

The TASC survey followed a relatively simple format and was broadly split between questions relating to teaching and questions relating to assessment. This page focuses on the assessment portion of the TASC (questions 18 to 37). Ten assessment ‘solutions’ were presented to respondents, starting with remote assessment methods (online written exams, vivas, portfolios of work, coursework), then delaying assessment (postponement, suspension), and finally alternative grading methods (on-course performance, grades already achieved, unclassified degrees).

For each the respondent was asked to rate the effect that implementing the measure would have on them, with 1 indicating an ‘overall negative effect’ and 5 indicating an ‘overall positive effect’. For each measure respondents were then asked to explain their rating and detail any specific effects that the measure would have on them. Oxford SU was especially concerned with the content of these written submissions and have drawn heavily from the most insightful and illustrative (anonymised) quotes when lobbying the University.

In the graphs below, the ratings given by students are provided alongside a breakdown of the concerns students raised in their written submissions. Many submissions raised multiple points and many students chose not to give a rating or not to accompany their rating with any additional explanation – hence why the numbers of ratings differ from the numbers of concerns raised by students.


Assessment via Online Written Exams (open book, 3 hours)

In the University’s initial communications with students, it indicated a desire to move to remote forms of assessment. This first proposal aims to keep the format of assessment as close to what was expected as possible (of course there are non-3 hour papers but this is a typical paper length). Online papers would have to nominally be ‘open-book’ as there would be no way to prevent students from accessing resources during remote exams. Nonetheless it was expected that keeping the exam time close to the normal length would mean the open-book format would make minimal difference in terms of how students would approach the paper. Although other universities have been announcing very large time windows (e.g. 24 hours and 48 hours), both to mitigate the disruptive impact of the ongoing crisis and to accommodate students sitting exams in different time zones, this comes with a risk to student welfare. It was noted that students may feel pressured to unnecessarily use as much of the available time as possible which would have an especially terrible impact on students sitting exams on consecutive days.

Many students felt that the overall impact on them would be negative, with many students citing home circumstances unsuitable for sitting remote exams. It was very clear from these results that the University needed to do more to understand specific home circumstances and colleges have now distributed a survey tailored to achieve this. Students’ differing home circumstances also mean they have unequal access to educational resources which would disadvantage students both in preparing for an exam and also during an ‘open-book’ exam. Furthermore, there was scepticism as to whether certain papers could be suitably adapted to an online format. Where this approach was viewed positively, it was often so viewed because of the clear effort being made to make the exams as close to the expected format as possible.


Assessment via Online Written Exams (open book, 5 hours)

A larger exam window of 5 hours would accommodate technical issues that might arise for students being examined in a new format for the first time. It would also allow students to take advantage of the ‘open-book’ format.

Again, students regularly referred to the need to keep exams as similar to the normal format as possible and 5 hour exams were criticised on these grounds as well as on the basis of students’ unequal home conditions. Notably, a longer exam requires a student to have access to an exam-suitable environment for a longer time.


Assessment via Online Vivas

Replacing exams with vivas would have been a radical departure from students’ expectations, however students indicated that such a form of assessment should exist optionally in an open letter to the Pro-Vice Chancellor (Education). It was also suggested in the earlier Cambridge open letter and was included here on that basis.

Perhaps unsurprisingly, vivas were more widely seen as having negative effects than e-exams. Vivas require students to have more sophisticated equipment than online exams and technical issues and unequal home circumstances were once again raised. The predominant concern was with the unfamiliarity of the format. Online exams have raised additional concerns regarding cheating and, as ever, to deter cheating, the University reserves the right to use viva voce exams to check candidates understanding of examined material but this has never formed a large part of the assessment strategy.


Assessment via Portfolios of Tutorial-Style Work

Assessing students via a method they understand and are familiar with is essential for basic fairness. Students, by and large, have been preparing for formal written exams however they are also familiar with tutorial-style work (for example essays and problem sheets). It was thus posited that asking students to submit and be graded on a portfolio of work similar to that which they have already completed over the course of their degree could be a compromise. This would effectively be a coursework submission but would probably have less stringent guidelines than coursework typically has.

This proposal was significantly more favoured by students however many miscellaneous issues were raised regarding specific courses and student circumstances. It was recognised that this form, though a radical departure for the University, would involve a form of work familiar to students and those who felt it would affect them positively wrote that it would mitigate the issue of home circumstances unsuitable for sitting formal timed exams. There were concerns that the expected standard would be very high compared to work students produce under strict exam conditions and several pointed out that due to the quantity of exam papers they had anticipated sitting in Trinity portfolios would have to be very large to adequately replace the papers.


Assessment via Coursework Assignments

Another form of assessment that many students are familiar with is formal coursework assignments. These would address the issue of familiarity at least for those students who have submitted coursework before, as well as being a widely accepted way of assessing students for their final grade.

Oxford’s courses display a great deal of diversity. Whilst many favoured this approach there were serious concerns about whether it could be applied to specific courses. Students consistently observed that completing coursework would likely require access to resources they could not be confident that they would have to hand. Coursework also requires a large set window in which to complete and if, for example, each paper a student was going to sit were to be replaced with an extended piece of coursework it could result in an intolerable and unprecedented workload burden. Departments have been empowered to replace individual papers as they see fit with coursework assignments.


Remote Exams Postponed to End-of-Summer

If the major difficulty for students was a lack of time to adapt to unfamiliar exam formats, then this could be addressed by some form of postponement. However, without guarantees that the pandemic would abate soon, postponed exams would still be in the online format.

Many students detailed the various individual plans and commitments that made such a delay deeply troubling for them. Students were also concerned that, given the fundamental issues with remote assessment would not be addressed and given the pandemic would likely be on going, this measure would merely prolong this stressful process. The ramifications of extending exams into an uncertain future weighed heavily on the decision-making processes. A core principle of the University has been the desire to ensure that Final Year Students especially are able to satisfactorily complete their degrees in a timely fashion.


Students Supported to Suspend their Studies

Many students have expressed a desire to suspend their studies and return in the next academic year. This happening on a large-scale is a logistically undesirable outcome for the University. It is also not a decision for any individual student to take lightly. It was important to understand student concerns relating to suspension.

It is clear that there is widespread desire for the option to at least be available for students, even if it is practically impossible for many. In response, the University have announced a second sitting of Trinity 2020 exams in Trinity 2021. These will take the traditional format and students will be able to pursue this option with the blessing of their college.


Grading Based on On-Course Performance

A view taken by some has been that any form of fair assessment is unviable in Trinity 2020 given the circumstances. Thus, several possibilities for grading without assessment have been discussed. The concept of grading students of the basis of their performance on-course thus far has formed the basis of several university’s ‘No Detriment’ policies (policies to ensure that students’ grades will in no way be disadvantaged by the ongoing pandemic). At Oxford, students typically undergo a relatively small amount of summative assessment before their final year. This means on-course performance would, in many cases, be measured by, for example, OxCORT reports and submissions from tutors.

This was a divisive option with large numbers of students asserting it would be fairer than proposed Trinity assessment methods and many others arguing that it would be categorically unfair. Whilst students’ performance on-course does give an indication of their ability and could well predict how they would be likely to perform in exams the point was frequently made that students were not expecting to be assessed on the basis of formative assessment. It was widely recognised that informal assessment by tutors is open to large biases and several postgraduate students who are themselves tutors expressed deep concern with this form of grading.


Grading Based on Grades Already Achieved

Whilst Oxford students generally complete relatively little summative assessment until their final year many students will have achieved formal final grades in, for example, coursework and dissertation pieces. These grades could be used to determine the grade that a student deserves based not on submissions from colleges or tutors but on work that students did in the clear expectation of being formally graded.

Many students highlighted how this approach could not apply to them as their course had no or very limited coursework elements. Many students also expressed a desire to improve on their performance in assessments already sat. Nonetheless there was a strong steer that grades already achieved are a fair representation of a student’s performance. Grades already achieved will form the foundation of the Safety Net policy for many students. This policy keeps open the possibility of students improving their overall grade based on their exam performance.


Option to Graduate with an Unclassified Degree

Finally, we asked students about the possibility of graduating with a ‘declared to have deserved honours’ (DDH) degree. This concept also draws from the idea that it is not possible to fairly assess the current final year cohort. DDH would be a special unclassified degree that students would have the option to take instead of sitting any further assessment. This would allow final year students for whom further assessment was unviable to graduate and move on to the next stage of their lives.

In the question it was made clear that, unlike some of the other proposals, this would be optional for each student. Many students felt therefore that it would not especially affect them, as they would not choose this option if it were made available. There was a steer from students that the option should at least exist but there was deep concern as to what an unclassified degree would mean for the students who took it in terms of employment prospects and future study. Many final year students have job offers and offers for further study that require achieving a specified classification (e.g. Undergraduate 2.i). The University have elected to make DDH an option. In order to mitigate the potential negative effects for students who take this option (and conscious of the fact that these will be the students worst affected by the current pandemic) DDH degrees will be accompanied with an enhanced reference letter from the University, indicating the classification that the student would have been working towards.


The above gives an insight into the findings of the TASC, and the Student Union’s ongoing work with the University on academic affairs during this difficult period. The SU will be releasing a similar summary focusing on the arrangements for remote teaching shortly.

If you have any questions or would like the SU’s support on an issue of access to education or academic representation and policy, you can contact the Vice-President for Access and Academic Affairs here: vpaccaff@oxfordsu.ox.ac.uk