When I work with instructional designers and K-12 teachers, I sometimes ask them, “How do they know in what order to put the choices in a multiple choice questions?” Very often the answer is something like, “Well, I’ve had enough Bs already, so the answer needs to be a C this time.” In other words, they are attempting to make the choices and correct answer random. But in reality, it won’t be. Another point about this is that the teacher or trainer feels like he or she is consciously trying to determine the ordering of the distractors and the correct answer (key). I find when trainers and teachers do this, they inadvertently contribute to the cognitive load of the learner. In other words, I believe, they are unintentionally putting an unnecessary layer between the learner and the learning. When instructional designers and K-12 teachers try to “mix up” the answers, they add extraneous cognitive load to the learner, because the learner will think about the order of the choices first.

Key and distractors in multiple choice items

Key and distractors in multiple choice items

In my last post, I discussed how poor designs and navigations can certainly contribute to the cognitive load of learners.  In Part 2, I’d like to spend some time considering the cognitive load of objective assessments.

There are a couple of different purposes for assessment.  Probably, first and foremost, we are interested in determining what the learner has learned.  Secondly and inherently related to the first purpose, we are interested in differentiating between the learners who know the correct answer and those who are guessing.  (This is often referred to as discrimination.)  Lastly, we are often interested in our learners learning from the assessment itself.  For example, sometimes we are asking learners to combine pieces of information or learning into a way that they may have not considered before or solving a novel problem that may not have been directly covered in the instruction. In K-12 and higher education but atypical to training situations, we are sometimes interested in manipulating the difficulty of items to further increase the discrimination among learners.

Jon Mueller on his site lists a number of techniques to create good multiple choice items.  Like Jon, I’m recommending you put the multiple choice options (Is that redundant?) in a logical order.  In addition, Haladyna, Downing and Rodriguez (2002) in the journal of Applied Measurement in Education also suggest using a logical or numerical ordering of distractors and key, which was particularly important to lower ability students.  In both training and K-12, this could be significant for low-level or struggling readers.  So, here’s three strategies for you to use.

1. Use alphabetical order

When the key and distractors are text, then order them alphabetically. In the example below, three of the distractors are very similar, so they are ordered alphabetically. The fourth option is different and so was separated to make it more noticeable.

Use numerical order for dates and numbers.

Use numerical order for dates and numbers.

2. Use ascending or descending order

When the key and distractors are numerical or dates, then use either ascending or descending order.  While there isn’t any research or recommendations that I found that suggested you stick with one throughout the entire assessment, I think you should.  But it probably doesn’t matter a whole lot if you don’t.

Use numerical order for dates and numbers.

Use numerical order for dates and numbers.

3. Use an existing logical pattern

Many times the content has a specific or logical pattern, such as with the order of a process, the order of Henry VIII’s wives or even the frequency of energy in the electromagnetic spectrum.  In these instances when the content dictates a logical pattern, then use the existing pattern.  However, an instance where you would not follow this recommendation is when you want the learner to list or rank the order.

If a pattern exists, use the logical pattern.

If a pattern exists, use the logical pattern.

By using these strategies, you are reducing the cognitive load.  Learners are no longer attempting to figure out the pattern, then figure out the correct answer.  Moreover, you are also taking the “control” of the correct answer placement out of the judgement of the trainer or teacher.  Instead, the key lands wherever it lands in the pattern, alphabetical or otherwise.

So, the answer that “I’ve already had enough Bs” isn’t one you should be distracted by anymore.

Enhanced by Zemanta

About Michael M Grant

Dr. Michael M. Grant is a passionate professor, researcher, and consultant. He works with faculty members, schools and universities, and districts to integrate technology meaningfully and improve teaching and learning. When 140 characters just won't work, then he blogs here at Viral-Notebook.com. He has a beautiful wife and three equally beautiful daughters, who will change the world.

12 Thoughts on “Increasing cognitive load without even knowing it: Part 2

  1. Chuck Hodges on May 6, 2009 at 4:57 am said:

    Thanks for this posting Michael! I work in a lab, the Virginia Tech Math Emporium, that delivers over a million (really) multiple-choice assessments each semester. This gives me things to ponder.

    Cognitive load for our math students is something I often think about, but I have not done any research on the issue. As our students complete their unproctored math assessments they can be observed listening to music, chatting with AIM off and on, and surfing the web, among other activities. Does their split attention impact their learning? I would love to know! So many questions, so little time…..

    • admin on May 6, 2009 at 6:42 am said:

      Chuck, I know that many of our course management systems allow randomization of questions and even the key and distractors. I feel like the second randomizing option here is not a good idea, but I do not believe there’s been extensive research on it either. In addition, our course management system here through a server side plug-in will allow us to lockdown the browser. Do you think that would be of any benefit to your students?

  2. Chuck Hodges on May 6, 2009 at 7:17 am said:

    If by “locking[ing] down the browser” you mean preventing students from visiting other sites while in a test, I do not think it would help very much, unless used with other software tools to limit the use of other applications too. Right now they use iTunes, iChat, etc. in addition to the browser running the quiz.

    Our quizzes are unproctored assessments, so if we locked them out on our systems, they could take them with their laptops or other computers. Perhaps the current generation of students can focus one question at a time, ignoring other things on the screen. In that case, maybe cognitive load from all of these things going on is not an issue.

    These ideas will definitely go in my notebook of research ideas!

  3. Cassie (UUG) on September 27, 2009 at 2:26 pm said:

    As a teacher, I am frequently creating multiple choice tests for my students. I have always wondered how I decide the order of the possible answers. After reading your page, I am going to use your ideas the next time I make a multiple choice test. For example, I thought the idea of putting the possible answer choices in alphabetical order would relieve some of the stress that the students feel when I hand out a multiple choice test. The students could rule out the answers that look similar, or alike. In addition, I would really like to use ascending and descending order on the next multiple choice test. I usually try to arrange the numbers randomly on the test. Using the ascending and descending order would give me a constant system for creating a multiple choice test. I thought these ideas were beneficial because I definitely do not want the students to have a cognitive overload.

  4. Amy H. (UUG) on September 28, 2009 at 4:54 pm said:

    I find this article very interesting. I teach kindergarten so I do not make many multiple choice tests. I do most of my assessing through observing and oral testing. I clearly recall the days of taking such tests and remember trying to figure out a pattern. My rule was when in doubt, choose C. The suggestions you give for making tests less stressful for students are wonderful. I would like to share this article with my own kids’ teachers!

  5. Tiffany M. (UU) on November 5, 2009 at 11:20 am said:

    Thanks for posting this interesting article! I am a Union University pre-service teacher working towards a degree in English Education. When taking multiple choice tests, I often look for patterns within the multiple choices myself, as well as patterns on the key. While I usually do well on these types of assessments, I often find multiple choice tests to be harder than short answer or essay. I experienced some of the frustrations of over-thinking just this morning on an all multiple-choice exam. When I am a teacher, I want to make sure that I do not subject my students to cognitive overload on tests I have created. I do want to test their knowledge/understanding of the concepts, but I do not want to stress them with the exam itself any more than is necessary. As a teacher, I believe it will be my job to give every student the best chance to succeed that is within my power.

  6. I find this blog post very interesting. I am studying to become a High School teacher, and this is definitely one of those things that you need to know when going into the teaching profesison. Clearly multiple choice tests are the simplest and most effi ent type of test to grade and assess. I want the tests that I give to my students to be effective in assessing what they have learned. I like that these suggestions make the tests less stressful for the students, which will again allow them to complete the test to the best of their knowledge.

  7. Amanda (UU) on November 10, 2009 at 7:02 am said:

    I really enjoyed this article. I am also going to school to become a teacher. I find myself now looking for the correct answers, but I never thought of something being a distractor, which makes since now, being that there is only one correct answer in a multiple choice question. Also, this allows the students to think more for just the correct answer not the pattern of the bubbles (which I do all the time). I found this article very helpful. Thanks for taking the time to post it, and share it!

  8. Victoria S. (UU) on March 8, 2010 at 6:20 pm said:

    This article was very interesting to me as I am currently in a class specifically designed to teach assessment. We just completed the chapter that discussed the uses of a multiple-choice test and how it should be designed; the idea of using logical order for distractors was not mentioned. My other textbook mentioned several times the need to design tests carefully so they would not become a “puzzle” for students, rather than a genuine form of assessment. I agree with my textbook and am very grateful for the practical advice of how to go about carefully designing such a test. The idea of reducing the cognitive load of a test has great appeal to me as I will be teaching in an ESL classroom; my students will already struggle with the format and language of the test, therefore reducing the cognitive load in whatever way possible will be a major goal in my assessment design.

  9. Pingback: Tweets that mention Viral Notebook | Michael M. Grant, Ph.D. -- Topsy.com

  10. Julie Zammit on September 18, 2013 at 8:17 pm said:

    Thanks for helping me look at not only my tests, but other national tests I use in my classroom I use in a new light. I’ll certainly be able to implement these excellent ideas

Post Navigation