When I work with instructional designers and K-12 teachers, I sometimes ask them, “How do they know in what order to put the choices in a multiple choice questions?” Very often the answer is something like, “Well, I’ve had enough Bs already, so the answer needs to be a C this time.” In other words, they are attempting to make the choices and correct answer random. But in reality, it won’t be. Another point about this is that the teacher or trainer feels like he or she is consciously trying to determine the ordering of the distractors and the correct answer (key). I find when trainers and teachers do this, they inadvertently contribute to the cognitive load of the learner. In other words, I believe, they are unintentionally putting an unnecessary layer between the learner and the learning. When instructional designers and K-12 teachers try to “mix up” the answers, they add extraneous cognitive load to the learner, because the learner will think about the order of the choices first.
In my last post, I discussed how poor designs and navigations can certainly contribute to the cognitive load of learners. In Part 2, I’d like to spend some time considering the cognitive load of objective assessments.
There are a couple of different purposes for assessment. Probably, first and foremost, we are interested in determining what the learner has learned. Secondly and inherently related to the first purpose, we are interested in differentiating between the learners who know the correct answer and those who are guessing. (This is often referred to as discrimination.) Lastly, we are often interested in our learners learning from the assessment itself. For example, sometimes we are asking learners to combine pieces of information or learning into a way that they may have not considered before or solving a novel problem that may not have been directly covered in the instruction. In K-12 and higher education but atypical to training situations, we are sometimes interested in manipulating the difficulty of items to further increase the discrimination among learners.
Jon Mueller on his site lists a number of techniques to create good multiple choice items. Like Jon, I’m recommending you put the multiple choice options (Is that redundant?) in a logical order. In addition, Haladyna, Downing and Rodriguez (2002) in the journal of Applied Measurement in Education also suggest using a logical or numerical ordering of distractors and key, which was particularly important to lower ability students. In both training and K-12, this could be significant for low-level or struggling readers. So, here’s three strategies for you to use.
1. Use alphabetical order
When the key and distractors are text, then order them alphabetically. In the example below, three of the distractors are very similar, so they are ordered alphabetically. The fourth option is different and so was separated to make it more noticeable.
2. Use ascending or descending order
When the key and distractors are numerical or dates, then use either ascending or descending order. While there isn’t any research or recommendations that I found that suggested you stick with one throughout the entire assessment, I think you should. But it probably doesn’t matter a whole lot if you don’t.
3. Use an existing logical pattern
Many times the content has a specific or logical pattern, such as with the order of a process, the order of Henry VIII’s wives or even the frequency of energy in the electromagnetic spectrum. In these instances when the content dictates a logical pattern, then use the existing pattern. However, an instance where you would not follow this recommendation is when you want the learner to list or rank the order.
By using these strategies, you are reducing the cognitive load. Learners are no longer attempting to figure out the pattern, then figure out the correct answer. Moreover, you are also taking the “control” of the correct answer placement out of the judgement of the trainer or teacher. Instead, the key lands wherever it lands in the pattern, alphabetical or otherwise.
So, the answer that “I’ve already had enough Bs” isn’t one you should be distracted by anymore.