In Part 1 we suggested that if AI can answer many conventional exam questions, it’s time to take a different approach to the questions we ask. But there’s another aspect to this question we need to consider: should we always be trying to keep AI out of assessment?
There are obvious reasons why we might. After all, there are many things we want medical students to know and be able to do for themselves. Importantly, we want to know that they have a clear grasp of the fundamentals. For example, that they can recognise a deteriorating patient, interpret an ECG, work out an appropriate drug dose or make a clinical decision, without simply relying on a machine for the answer.
However, by the same token, the students we are assessing will graduate into a world and a healthcare system in which AI is likely to be part of everyday professional practice. Healthcare professionals need to know how to practise without it, but also how to use AI appropriately. This suggests that “AI allowed” versus “AI prohibited” may be too simple a distinction.
Varying the approach to AI in assessment
There may be an opportunity for different assessments to have different relationships with AI. In some assessments, AI might quite reasonably be prohibited because the purpose is to establish what the student can do independently. In others, AI might be permitted because use of technology is part of the skillset we want to assess. Moreover, there may be assessments where AI is deliberately introduced as part of the task.
For example, what if students were given an AI-generated clinical answer and asked to assess it: is it correct? What has it missed? Is anything unsafe? What evidence would they want to check? Would they follow the recommendation for this particular patient?
This also reflects an important aspect of using AI in clinical practice. The challenge isn’t necessarily to produce a better answer than the AI. It may be to recognise when the answer is useful, when it needs checking, and when/if it is simply wrong. This raises some interesting possibilities for digital assessment. First, the exam might ask the student to work through a clinical case independently. Then provide an AI-generated interpretation and ask the student to critique it. To identify what they agree with, what they don’t, what they would verify, and what they would ultimately do for the patient. In this scenario, the technology is no longer something outside the examination that students are trying to use secretly but becomes part of the assessment itself.
Critical appraisal and clinical judgement
Essentially, our approach should be balanced. We shouldn’t assume that because AI is becoming ubiquitous, students no longer need to demonstrate knowledge and reasoning without it. Nor should we make every assessment artificially difficult by removing tools that students will routinely use in practice. The important question is not simply: “Can students use AI in this exam?” but, more to the point: “What are we trying to find out about the student?” This can lead to three rather different questions: as a medical practitioner, Whatmust I be able to do myself? What can I appropriately use AI to help me do? How do I know when the AI is wrong? The third may turn out to be particularly important. A doctor who accepts an AI-generated answer uncritically could be dangerous. But a doctor who refuses to use a potentially useful tool because it is AI may be working less effectively than they could.Medical education therefore has to prepare students for both situations.
The future of assessment may be less about deciding whether AI is in or out, and more about becoming much clearer regarding what belongs to the student, what can appropriately be supported by technology, and whether the student has the judgement to tell the difference. AI may be changing not only how we assess students, but what we think competence means.