When developing a medical or dental examination, it is likely that you will want to include different types of questions:
Essential knowledge questions
These are more basic questions, usually worth less marks, to test that students know the fundamentals. They generally don’t take too long to answer.
Ranking questions
These are harder questions, usually worth more marks, which help discriminate between students to help determine the best students.
However, many examination marking schemes in our experience do not discriminate between the different types of questions when determining who passes and fails, as that decision is made on the total marks achieved. This raises some concerns as we lay out below.
Lies, damn lies and statistics
In the example above, two candidates have sat the same exam and achieved the same number of marks by answering different questions correctly, but assuming a pass mark of 3/6, should both students be allowed to pass?
Our view would be that while answering the ranking question correctly is desirable, it is essential that the student has a good grasp of the fundamentals.
This is of course a simplified example, but the same principles apply in a longer examination. When essential knowledge and ranking questions are not separated when a paper is being marked, there is a real risk that students may pass an exam without the foundation knowledge required to make them a good doctor or dentist.
Ensuring your exams identify both the safe and the best candidates
We would suggest that passing an exam should not just be based on an overall mark, but instead should also depend on the candidate having got a high percentage of the essential knowledge questions right (e.g. 90% of these), with the ranking questions used to identify the stronger and weaker students among those who have demonstrated they know the basics. In other words, even if a student was to reach the overall pass mark in an exam, if they did not reach the required marks for the essential knowledge questions then they would fail the exam.
The above however does depend on being sure up front what your questions are trying to achieve – i.e. are they measuring essential knowledge or ranking candidates. Once this decision is made, this kind of analysis is something that Maxexam offers our customers as part of the software, and we can advise how best to take advantage of it.
Standard setting for success
If the exam is to achieve what it set out to do, it also relies on the questions being standard set correctly – i.e. if an essential knowledge question should be answerable by 90% of the students sitting an exam, you need to be sure that the question is set at the right level and also check that actual results support this.
This means that items/questions need to be analysed post exam, to check they are achieving what they are supposed to. Measuring item reliability to ensure that questions are achieving what they are supposed to will be the topic of our next blog.
In conclusion
I’m sure we would all agree that it is vital that doctors and dentists have the right level of knowledge to treat us, their patients, properly.
If doctors and dentists are to be ‘safe to practise’ then we think it is vital that exams and marking schemes ensure that all passing candidates meet the grade for the essential knowledge questions, with the ranking questions being used to identify the best students among those who demonstrate this knowledge.
If you have any comments or would like to discuss this further with us, please pick up the phone or drop us a note through our contact form.
