Imagine two medical students with essentially the same knowledge. They know the relevant physiology. They recognise the important clinical features. They can identify the appropriate investigation and treatment. But one reaches the answer in 20 seconds. The other takes 50 seconds. In a tightly timed exam, that difference may affect their score. So what have we actually measured? Knowledge? Clinical reasoning? Speed? Or some combination of all three? This is not simply a question about exam technique. It goes to the heart of what an assessment score means.
Medical exams have always been subject to time constraints. But digital assessment gives us an opportunity to think more carefully about the relationship between what we want to measure and the conditions under which we ask candidates to demonstrate it.
A digital exam can reproduce a paper exam almost exactly. But it can also do much more. It can present information dynamically, incorporate images, audio and video, require candidates to generate rather than recognise answers, and structure clinical problems as sequences of decisions. The technology gives us more choices. The challenge is deciding which choices make sense for the capability we are trying to assess.
- Knowledge: what does the candidate know? Medical education necessarily depends on knowledge. Doctors need to know anatomy, physiology, pharmacology, pathology, diagnostic criteria and treatment principles. Digital exams can assess this efficiently, with large numbers of questions, consistent presentation and automated marking. But even apparently straightforward knowledge questions can test rather different things.
- Compare: Which antibiotic is recommended for this infection? With: A patient presents with these symptoms, examination findings and laboratory results. Which antibiotic is most appropriate?
Both could be single-best-answer (SBA) questions. Yet the second asks the candidate to apply knowledge to a clinical situation. The important pointer is that question format does not determine cognitive demand. An SBA can test recall, application or sophisticated reasoning. Simply because an exam is delivered digitally does not tell us what it actually measures.
- Reasoning: what can the candidate do with what they know?
Clinical practice requires more than remembering facts. Doctors need to interpret information, recognise patterns, generate possibilities, decide what further information is needed and make decisions. Digital assessment potentially allows us to represent these tasks in richer ways. A clinical case might include an ECG, radiograph, laboratory results, audio or video. Information might be revealed sequentially, requiring the candidate to interpret each stage before deciding what to do next. This can provide evidence about clinical reasoning that a static recall question cannot. But there is a danger in assuming that greater technological sophistication automatically means better assessment. The object is not to make an exam more elaborate or more “realistic” simply because technology allows it.
The objective is to collect better evidence about the capability we are trying to assess.
- Speed: how quickly should knowledge be demonstrated?
Every timed examination contains an element of speed. But speed can play very different roles. Sometimes the time limit is simply an administrative necessity. We need the exam to finish within a reasonable period, but speed is not itself part of the capability being assessed.In other assessments, speed is deliberately part of the task.
Psychometrics has long distinguished between power tests, where candidates are given sufficient time to demonstrate what they know and can do, and speed tests, where rapidperformance is central. Many medical examinations sit somewhere between the two: they are essentially speeded-power tests. This raises a question: How much of a candidate’s performance reflects what they know, and how much reflects how quickly they can demonstrate it? That doesn’t mean time pressure is necessarily inappropriate. In some clinical situations, rapid decision-making is essential. But if speed is not part of the capability we intend to assess, we need to be clear about the function of a particular time constraint. The interaction matters.
Knowledge, reasoning and speed cannot always be separated neatly. In a tightly timed exam a candidate who processes the relevant information faster may attain a higher score than a candidate who takes longer to interpret a complex case. But what does that difference tell us?
It may reflect knowledge. It may reflect reasoning. It may reflect speed. Or it may reflect all three. That matters because the exam score will be used to make consequential decisions. If we’re going to interpret a score as evidence of competence, we need to understand what contributed to that score.
What does digital change?
Digital assessment gives us an opportunity to be more deliberate about these choices.
We can decide what information candidates receive, how they interact with it, how they respond and how time operates. We can ask candidates to recognise an answer or generate one. We can present information simultaneously or sequentially. We can use text alone or incorporate clinical images, sound and video.
These are assessment decisions, not simply technical decisions. The starting point should therefore be the capability we want to assess:
Do we want to know what the candidate remembers? Can they apply that knowledge? Can they reason through an unfamiliar clinical problem? Can they make an appropriate decision under time pressure?
The answers should determine the assessment task. Only then should we consider which digital technology best supports it. A different starting point.
Perhaps, then, the question is not:
How do we put our medical examination online?
but:
What evidence do we need from our candidates, and what is the best task for obtaining it?
The real potential of digital technology is that it gives us more ways to align the capability, the task, the conditions and the evidence. And that’s a much more interesting opportunity than simply putting an exam on a screen.
It also leaves some important questions for further consideration: How should we choose between different question types? When does time pressure improve an assessment — and when does it distort it? And how can digital technology allow us to assess clinical reasoning rather than simply knowledge?
We will explore these questions in future posts.