2020 has been a year of enormous upheaval for healthcare educators. While many have been running their exams digitally for years in invigilated halls or OSCE centres, the arrival of Covid-19 and associated lockdowns has forced organisations to look at options to run medical and dental exams remotely.
We need doctors and dentists more than ever during these difficult times, and students still need to be able to complete their education and graduate in a way that ensures that they are safe to practise.
As a result, many universities have turned to running their written exams on a distributed basis (i.e. in students’ own homes) using remote invigilation – often referred to by the American term ‘Proctoring’.
What is Remote Invigilation/Proctoring?
From now on we will refer to this as Remote Invigilation. It describes using technology to invigilate exams without physically being in the room.
There are two stages of remote invigilation, both of which can be completed in different ways:
Stage 1 – Authentication
This stage confirms that you have the right student sitting the exam! This is done in one of two ways:
Auto authentication – Before the exam starts the student takes a photo (usually using their webcam) of their face and their photo ID and uploads these to be matched against their ID credentials on file. The student will then answer a few challenge questions and enter a biometric keystroke signature (typically the student’s first and last name).
Live authentication – The same process is followed as for auto authentication, but then a ‘live’ invigilator does a facial comparison using the student’s webcam.
Stage 2 – Invigilation
After the student is authenticated, they start the actual exam and are monitored in one of three main ways:
Automatic invigilation – The examinee’s surroundings are monitored during the exam using software to pick up sound, motion and systemic changes that indicate possible collusion. No-one actually ‘sees’ the student themselves using this method so it is less invasive of privacy.
Live invigilation – After completing authentication, the student and their surroundings are monitored by a live invigilator utilising the device’s webcam. This enables them to pick up on any infractions as they occur. This method of remote invigilation is the most comprehensive, however it is expensive and has potential to lead to exam delays due to technical problems, or key decision makers such as lead examiners being caught up elsewhere. There is also a question of privacy with examiners seeing into people’s homes and the potential for unconscious bias by the examiner driven by the student’s appearance or surroundings.
Record-and-review invigilation – After completing authentication the examinee is videotaped from the start to the finish of the exam and an invigilator later reviews the video. This method lies somewhere between the two methods above, being cheaper than live invigilation but more comprehensive than automatic invigilation. It does however share the same privacy concerns as live invigilation and there is the added question of what happens to the videos once they have been reviewed.
Can you just run the same exams online using remote invigilation?
This is the approach that has been taken by some universities who essentially are trying to mimic a traditional exam using technology. But is this the best option?
We would suggest that consideration needs to be given to how the exams will work most effectively if they are run remotely, as well as how to take full advantage of the options that technology opens up.
When examining organisations try to simply replicate their usual exams, this approach can lead to problems as The Bar Standards Board found out when they tried to run their usual law exams remotely using live invigilation. If you want a cautionary tale, you can read about what happened here, but suffice to say things didn’t go well!
Things to consider when running exams remotely
When exams are to be run remotely there are a number of factors that can influence the success or otherwise of the exam. The approach taken will depend on the exam itself but things to consider include:
Exam Structure
Depending on the exam itself, options to consider are:
Open Vs Closed book.
A closed book exam tests a student’s memory of facts as well as their application, whilst an open book exam is about testing their understanding and application of knowledge rather than the student’s memory. Clearly there is more incentive to cheat with a closed book exam where access to notes otherwise banned may provide a benefit.
More questions in less time Vs less questions in more time
More questions in less time will clearly reduce the opportunity for students to cheat by spending time looking up the answer, which might be applicable for exams using multiple choice. Less questions in more time, perhaps in combination with an open book exam, will allow students to more fully demonstrate their understanding of a topic which is likely to mean the better students rise to the top.
Exam objectives
Healthcare exams have a different dimension beyond simply who are the best candidates, in particular for qualifying exams which need to also identify who is ‘Fit to Practice’. This means that the approach for the high stakes qualifying exams will probably need to be different to that for earlier exams within a degree course. If you need to be able to identify who knows enough to be safe to start work as a doctor or dentist, then it is likely that open book exams may not be relevant and it’s imperative that the opportunity to cheat is reduced as much as humanly possible.
Technology
Technology can help to ensure fairer exams in 2 complementary ways:
Making it more difficult to cheat
Utilising the exam set-up options available within exam software can make it harder for students to cheat or collude with each other. For example, question randomisation means any two students are less likely to be on the same question at the same time. In combination with ‘forward only’ exams and perhaps limited time allowed on questions this can make it much harder for students to collude.
Identifying collusion
Exam software can be used to identify possible collusion using options such as IP address matching and cheat analysis (using the Harpp-Hogan Index).
Parity of Delivery
It is vital to consider whether all students have the means to run the exam equally. While most students nowadays will have some kind of device, not everyone will have access to a strong stable internet connection at their student accommodation.
Consideration therefore needs to be given both to how the exam is sat and the invigilation method employed to ensure that some students aren’t likely to be unfairly impacted by technical issues – something that happened in the Bar Standards Board exam mentioned earlier
Live proctoring options do tend to require a strong stable internet connection, and some exam systems also require the exam to be sat over the internet. If parity of delivery is likely to be an issue then you may wish to consider a digital exam management system like Maxexam which allows the exam to be downloaded in advance, so doesn’t require a good internet connection for the duration of the exam. Similarly, if invigilation is required you may wish to consider other options to live invigilation such as record-and-review invigilation.
Is invigilation really required?
In some cases (e.g. for formative exams) it may be possible and is generally much less expensive to run exams remotely without remote invigilation tools. Sophisticated digital exam delivery systems in combination with context information (such as students having the same tutor) can pick up most cheating as well as helping to prevent it in the way exams are set up. Additionally, if students are informed in advance that cheat analysis will be undertaken, they are less likely to try and cheat in the first place. Running exams as open book can also reduce both the desire to and benefit from cheating.
If exams involve a small number of students there is also the possibility that students could sit the exam on one device whilst also being on Zoom or Microsoft Teams on another device (which many students will use anyway for online lessons), which would allow university staff to invigilate an exam, eliminating the need to invest in a costly remote invigilation service. Again, this would not be appropriate for large scale, high stakes exams, but could be for small groups of students sitting low stakes exams.
Is it a written or clinical exam?
Whilst running exams remotely is feasible for written exams, currently none of the remote exam delivery and invigilation options available are really suitable for running clinical exams. This is however something that we at Maxinity and many other organisations are working on.
Conclusion
There is no one ‘magic bullet’ to ensure that all written medical and dental exams can be run remotely and deliver robust, fair outcomes. It is more nuanced than that, and involves careful consideration not just of if and how the exam is invigilated, but also how the exam is best structured in the first place.
The best route forward for each individual organisation and exam will depend on a number of factors including type of exam, budget, what the objective of the exam is, how ‘high stakes’ the exam is, what technology the students sitting it have access to, privacy concerns and more.
By taking a strategic, structured approach and involving your digital exam provider from the start, you can give yourself the best opportunity to deliver fair, robust medical and dental exams at all levels.
