What is standard setting?
Standard setting is the process used to distinguish between a competent student and an incompetent student. This is achieved by determining the score or mark associated with the minimal level of skill or knowledge required to reach a certain level of achievement. This line in the sand or “cut-off mark” divides the students in to two groups; those below the cut off and those above the cut-off. You may have an examination where there are two levels of achievement “pass” and “fail”. In this case, the cut-off mark would be the mark that a candidate must achieve or exceed to be awarded a “pass”. In other cases, the candidates may be placed into a greater number of achievement levels based on a scale. For example, “developing, proficient, advanced”.
Standard setting is a crucial part of quality assuring your exams. If the cut-score is incorrect then the exam results will be affected which could have huge implications. This is especially critical in high stakes examinations such as medical or dental where the results could mean the difference between becoming a doctor or not.
How do I choose a standard setting method?
There are many different standard setting methods that can be used to calculate the cut-off mark for an assessment, some more appropriate than others for different types of assessment. There are two main categories of methods used to calculate a cut-off mark:
Norm-referenced (Relative) – These are methods that are used to calculate cut-off marks where the number of passing candidates is relative to the rest of the candidates taking the exam. For example, the worst performing 25% of candidates will fail the exam. Relative methods are useful in situations when there are a limited number of places available such as entrance exams. They are not widely used for high stakes examinations as this process doesn’t actually define how well a candidate performed in an exam, just how that candidate performed in relation to the other candidates.
Criterion-referenced (Absolute) – These are standard setting methods where the cut-off mark is calculated based on the performance of candidates in relation to a defined standard as opposed to in relation to each other. This means that any number of candidates could pass or fail the exam, depending on whether or not they meet the minimum defined standard for the assessment. Examples include Angoff and Borderline regression.
Compromise – These methods are considered to combine elements of both absolute and relative methods. Examples include Hofstee and Cohen.
Within these categories there are two main types of standard setting methods:
Examinee centred – These standard setting methods use real exam data to calculate a cut-off mark., based on an examiner’s judgement during the exam. These methods involve a judgement being made of an exam candidate (examinee) as opposed to an exam item. For example, the Constrasting group method.
Test centred – These methods involve a cut-off mark being calculated before the exam takes place, based on the judgement of a group of subject experts. These methods involve a judgement being made of an exam item as opposed to an exam candidate. For example, Ebel.
Coming soon…
In this series of blogs, we are going to put a spotlight on some of the most popular absolute standard setting methods used in written and practical exams, and look at the pros and cons of applying these methods.
Look out shortly for the first in the series on “Absolute” methods, Borderline regression.
