What is item analysis?
Item analysis uses different statistical measures in order to determine any problems with the ‘items’ that make up a question in an exam, and hence the question itself.
Why is item analysis useful?
Let’s say that we have an item that has been standard set at 90%. This means that whoever made this judgement has suggested that the probability of getting this item correct is 90%, so prior to the exam we are going to expect candidates to do very well on the whole on this particular item.
The exam is then sat and turns out the mean for this item is down at 50% – only half of the candidates managed to get it correct. Why?
It could be a problem with:
1. The item
It may have been badly written and become misleading, resulting in candidates either getting it wrong or taking a long time on it.
The distractors (other options that are marked as incorrect) could have also been considered correct, so lots of candidates chose these options and got ‘distracted’ unfairly from the correct option.
The wrong option may have been selected as correct in the system whilst writing the item. Therefore, it appears the item is poorly performing.
2. The teaching
What’s to say that there isn’t actually a problem with the item and that the candidates were simply not taught it in class? It may be a well written item, but if the material wasn’t taught in the classroom, how are you going to achieve such a high mean i.e. at 90%?
Item analysis means we can take a deeper look into what’s happening on the particular items within a question, and hence work out if there is a problem with the questions themselves or with the teaching. It can also help to define why an item, and hence a question isn’t performing as expected.
What’s to say that there isn’t actually a problem with the item and that the candidates were simply not taught it in class? It may be a well written item, but if the material wasn’t taught in the classroom, how are you going to achieve such a high mean i.e. at 90%?
Item analysis means we can take a deeper look into what’s happening on the particular items within a question, and hence work out if there is a problem with the questions themselves or with the teaching. It can also help to define why an item, and hence a question isn’t performing as expected.
Working out why an item isn’t operating as expected
Statistical measures
Maxexam enables the easy use of statistical methods such as the discrimination index, Pearson’s product-moment correlation coefficient and Horst which can help to identify how different items are performing within an exam. So, for example, the discrimination index might show that the item is not as discriminating as expected between your high and low performing candidates.
These different statistical methods will be covered individually over the next few blogs.
Time taken on a question
Maxexam also enables the psychometrician to look at the time taken on items within a question. Looking at the aspect of time is an alternative approach to item analysis that can show why an item isn’t performing in an exam. It is possible that a question still looks like it’s performing to a high standard when looking at the usual statistics such as the mean, discrimination index, correlation etc, but it could then be the case that is has taken candidates on average lots of time to answer, having a knock-on effect on the rest of the exam.
Examples of when a question might take a long time to answer are if it has been worded in a way that is confusing or misleading, or if more than one of the answers given could be correct.
Without a digital solution like Maxexam, it is impossible to record the average time it took a candidate to answer a particular question, so these kinds of problems are less likely to be picked up.
In Summary
Identifying problematic items within an exam can be tricky and time consuming. Pushing that further and understanding the reason(s) why an item isn’t performing as expected can be even harder.
The more factors you can take into consideration the better, and the use of a digital solution like Maxexam makes these calculations robust, easy and quick. This means that problematic items can be removed from an exam if required to ensure the results are fair. Looking longer term, this also means questions and exams can be improved over time using the learnings from each exam run.
