What is Cohen?
Cohen is a method of establishing a cut-off score by taking 60% of the score achieved by the 95th percentile candidate. This simple and inexpensive method is described as a compromise method, as it has elements of both absolute methods (based on the performance of candidates in relation to a defined standard) and relative methods (where the number of passing candidates is relative to the rest of the candidates taking the exam).
Why 60%?
The original Cohen method was proposed by Cohen-Schotanus and Van der Vleuten in 2010. They suggested that using 60% of the score from top performing candidates would be more effective than using an arbitrary percentage (most commonly, 60% is used as the cut-off score in Europe) as top performing candidates are the best representation of the highest score that could be achieved on an exam. Read more about their research here.
Why the 95th percentile?
Cohen and Van der Vleuten studied the cut-off scores and failure rates of the same tests given to first-year medical students at Maastricht University over a 9-year period. They looked at calculating the scores using 60% of the 95th percentile and 60% of the very top candidate and determined that the 95th percentile provided more consistent and acceptable cut-off scores and failure rates. This can be explained by the fact that candidates above the 95th Percentile – sometimes known as outliers – could get a score much higher than the 95th percentile (the class ‘genius’)– so skewing the cut-off score and not allowing enough candidates to pass.
Is Cohen considered to be a valid standard setting method?
The Cohen method is considered less valid than recognised absolute methods of standard setting as the only thing that separates it from an arbitrary cut-off score (for example, one standard deviation below the mean mark) is that it bases the cut-off score on the performance of the 95th percentile candidate. This assumes that the student at the 95th percentile is an accurate representation of the rest of the students’ ability and also that the student at the 95th percentile is consistently a good benchmark, and while there is some evidence supporting this it is far from conclusive.
How is Cohen calculated?
Cohen is calculated by taking the candidate in the 95th percentile (the candidate whose score is higher than 95% of the rest of the candidates taking the same exam) and finding 60% of their score. This is considered the cut-off score that a candidate would be expected to achieve to pass the exam. For example, if the total mark of the exam is 120 and the 95th percentile student got 92 marks, then the Cohen cut-off score would be 55/120 (46%) – which is 60% of 92.
The Cohen method and Multiple Choice Questions (MCQs)
Multiple choice questions by their nature can be guessed – e.g. if a multiple choice question has 5 possible answers, you would have a 20% chance of getting it right even if you had no knowledge of the subject at all.
This needs to be taken account of when calculating Cohen for an exam containing MCQ’s.
For example if you were to take the same total mark and 95th percentile candidate mark as in the previous example, but apply them to a multiple choice exam where each question has 5 possible answers, the calculation would need to be adjusted as follows to take account of the ‘chance percentage’ people would be expected to get right if they guessed:
As each question has 5 possible answers, the chance percentage of getting each question right is 20%
20% of 120 (the total marks possible) is 24, and this is the average score someone with no knowledge of the subject would get if they guessed all answers.
The 95th percentile candidate got 92 marks (77%) – so once the chance percentage is taken into account they got an additional 68 right (92-24).
60% of 68 is 40.8, meaning that the Cohen cut off in this case should be 40.8 plus the expected marks from guessing, so 64.8 marks out of 120 (54%). This is higher than the 55 marks (45.8%) calculated for an exam where guessing the correct answer was not possible.
Calculating the chance percentage
This will depend on the make up of the test. So for example if all questions in the exam had 8 possible answers then someone with no subject knowledge would only guess 1 in 8 questions so the chance percentage is 12.5%. The chance percentage can still be calculated when there are a mixture of question types, for example if there were 100 possible marks to be gained in a test and half of those could be gained from true/false questions and half from open ended questions, then the chance percentage would be 50% for the true/false questions and 0% for the open ended questions so the combine chance percentage would be 25%.
If you are using Maxexam to calculate the Cohen cut-off score, the software is now able to calculate the chance percentage for you, to ensure that it is entered correctly.
Should you use Cohen?
The Cohen method is very affordable and simple to apply and does have some substance behind the use of the 95th percentile candidate as a good representation of the best that could be achieved in the exam except by outliers. However, the evidence for using 60% of the 95th percentile candidate is limited, raising questions about the validity of the method. For these reasons, the Cohen method is ideal for use in in-house, informal examinations but not for high-stakes examinations that have implications on people’s careers.
Advantages of Cohen
Quick & cheap – Cohen is a very quick method of calculating a cut-off score and doesn’t require a panel of judges which can be difficult and expensive to arrange, making it ideal for informal in-house exams.
Simple – The maths behind the Cohen method is simple, you simply need to calculate 60% of the result of the 95th percentile student, and make amendments for guessing.
Reduced variability – Studies have found that Cohen consistently produces similar cut-off scores for the same exam over different years and cohorts of students.
Disadvantages of Cohen
Not ideal for high stakes – This method is not very valid or reliable for use in high stakes examinations that decide the fate of student’s careers or progression in education as 60% could be considered an arbitrary measurement.
Assumptions – This method assumes that the student in the 95th percentile is a consistent high performer, and also that they are a fair representation of the abilities of the rest of the cohort.
In summary, the Cohen method is a quick and simple method of calculating a cut-off score in low stakes, informal examinations. It would be advised to consider using a more justifiable method of standard setting such as Angoff or Borderline regression for high stakes examinations.
Coming soon…
Look out for the next blog in our “Standard setting simplified” series… Ebel.
