Imagine a group of medical students answering the same 50 questions. First, they select their answers from a list of options. Then, they answer the same questions by generating a response themselves. The subject matter has not changed, and neither have the questions. Only the response format has changed. Yet their scores are substantially different.
A study involving 1,417 students across 20 UK medical schools* found that mean scores were 21 percentage points higher in the single-best-answer (SBA) format than in the very-short-answer (VSA) format. The finding raises an important question: if changing the question format changes the score, what exactly does that score tell us about the candidate?
The task behind the question
It’s tempting to think of question formats as different ways of presenting the same task. In reality, each format asks candidates to do something slightly different. An SBA question requires the candidate to select the best answer from a list of alternatives. A VSA question requires them to generate an answer, usually in just a few words. One involves choosing; the other involves producing. The knowledge required may overlap considerably, but the cognitive task is not identical.
The distinction is often described as recognition versus recall. When answer options are provided, candidates may recognise the correct response even if they would struggle to retrieve it independently. The options can provide useful cues, help candidates eliminate implausible alternatives or allow them to make an informed guess. That does not make SBA questions inherently poor assessment tools. A well-designed SBA can test complex clinical knowledge and reasoning. But the answer options are part of the task, and their influence should not be ignored when interpreting the result.
What does the evidence tell us?
The UK study, published in BMJ Open in 2019, provides a useful illustration. Students completed a 50-question assessment first in VSA format and then in SBA format. The mean score was around 40% for VSAs and 61% for SBAs. The researchers also found that students frequently answered an SBA correctly after answering the equivalent VSA incorrectly, suggesting that the answer options provided more than a simple opportunity for random guessing.
There is an important qualification. Because students always completed the VSA questions first, they encountered the same questions again in the SBA paper. The authors acknowledged that this could have contributed to the higher SBA scores. The study therefore does not establish that answer options alone caused the entire difference.
Nevertheless, it highlights a fundamental issue: performance depends not only on what candidates know, but also on the task through which they are asked to demonstrate it. A score is evidence about a candidate’s performance under particular conditions. It is not a direct measurement of knowledge or competence, independent of those conditions.
Neither format tells the whole story
The implications extend beyond the distinction between recognition and recall. A short constructed response may reveal whether a candidate can independently retrieve a diagnosis, identify a relevant investigation or name an appropriate treatment. An SBA may be better suited to testing whether they can distinguish between plausible management options in a complex clinical scenario.
Neither task, on its own, necessarily establishes that the candidate can reason effectively in practice. A candidate might recall the correct diagnosis without being able to explain how the clinical findings support it. Equally, a candidate might demonstrate sophisticated reasoning through a well-designed SBA without having to write out every step of that reasoning.
The important distinction is between the format and the capability we want to assess. We should not assume that a particular question type automatically measures a particular level of thinking. A short-answer question can test straightforward recall; an SBA can demand substantial clinical reasoning. The design of the question, the information provided and the response required all help to determine the precise nature of the capability being tested.
There are also practical considerations. SBAs can be marked quickly and consistently, making them valuable for large-scale, high-stakes examinations. Constructed responses can provide different evidence, but may require more complex marking arrangements. Good assessment design involves understanding these trade-offs, rather than assuming that one format should replace another.
What does digital assessment change?
Digital delivery makes it increasingly practical to use a wider range of question types within an examination. Candidates might select an answer, generate a short response, interpret an image or work through a clinical scenario in which additional information is revealed progressively.
These possibilities are useful only if they serve a clear assessment purpose. Moving an SBA onto a screen does not change the underlying task. Equally, introducing a constructed response does not automatically make an assessment more authentic or valid. The opportunity lies in being more deliberate about the relationship between the capability, the task and the evidence we obtain. Digital technology can help us design and deliver different tasks, but it cannot decide which evidence is appropriate for a particular assessment decision.
Start with the evidence we need
When choosing a question format, it is worth asking what we need the candidate’s response to demonstrate. Do we want to know whether they can retrieve a key fact independently, distinguish between competing diagnoses, select an appropriate management plan, or interpret evolving clinical information?
The answer may lead us to different formats, or to a combination of them. It may also lead us to reconsider how we interpret scores when candidates have completed different types of tasks.
The question is not whether one format is universally better than another. Each creates a different set of conditions under which candidates demonstrate what they know and can do.
Before asking which question type to use, we should ask what we need the candidate’s answer to tell us.
*Sam et al. (2019) Comparing single-best-answer and very-short-answer questions for the assessment of applied medical knowledge in 20 UK medical schools: Cross-sectional study https://bmjopen.bmj.com/content/9/9/e032550